Home > Archive > PHP Programming > April 2006 > hitting the limits
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
hitting the limits
|
|
| windandwaves 2006-04-28, 9:58 pm |
| Hi Folk
I am managing a site, www.friars.co.nz that seems to be hitting the limits
and I keep getting the 500 error. According to the people from webfarm it
is because my script are too demanding or not closed properly.
I dont believe you have to "close" PHP scripts or even database connections.
The site also accesses a 80 Megabyte database.
Do you know of a way I can find out where the problems are or improving the
performance of the site?
I uses ob_start and ob_end_flush, would that cause problems?
TIA
> Nicolaas
| |
| Jerry Stuckle 2006-04-28, 9:58 pm |
| windandwaves wrote:
> Hi Folk
>
> I am managing a site, www.friars.co.nz that seems to be hitting the limits
> and I keep getting the 500 error. According to the people from webfarm it
> is because my script are too demanding or not closed properly.
>
> I dont believe you have to "close" PHP scripts or even database connections.
>
> The site also accesses a 80 Megabyte database.
>
> Do you know of a way I can find out where the problems are or improving the
> performance of the site?
>
> I uses ob_start and ob_end_flush, would that cause problems?
>
> TIA
>
>
>
>
>
Well, ob_start and ob_end_flush require system resources; the bigger the page
the more resources it requires.
You might not be using many resources with it. You may be using a lot. It all
depends on the site and what you're using it for.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
| |
|
| windandwaves wrote:
> I am managing a site, www.friars.co.nz that seems to be hitting the
> limits and I keep getting the 500 error. According to the people
> from webfarm it is because my script are too demanding
I'm no server admin, but maybe you can ask them what the most demanding
requests are?
> or not closed
> properly.
>
> I dont believe you have to "close" PHP scripts
Not to my knowledge, no.
> or even database
> connections.
Normally no.
If your resources are stretched, it may be worth it to check wether your
script opens once (and only once) a connection to the database per request,
extracts all needed data, and then closes the connection, after which is
continues further processing. If your website is that popular, maybe there
is something to gain from using a persistent database connection, that
highly depends on your used scripts and connections.
I might be talking out of my ass here, I've unfortunately never had the
problem of being so popular :-).
> The site also accesses a 80 Megabyte database.
Phah! That's huge compared to what I'm used to.
I assume you've normalized the database?
Created proper indexes for faster selecting?
> Do you know of a way I can find out where the problems are or
> improving the performance of the site?
To check how much time (and probably resource) your script takes, loop over
it a couple of times on a local server (somewhere in the 100 or more), and
on specific places in your code add the following:
At the start:
$start = microtime(true);
At key locations in the script you want to know the time taken.
$end = microtime(true);
$time_taken[$some_specific_name] += $end-$start;
$start = $end;
print_r($time_taken) after a 100 or so loops will make it clear to you which
portions of your code take the most time, and check those specific portions
wether they can be done more effective.
If PHP < 5 use www.php.net's suggestion: create a function:
function microtime_float()
{
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);
}
and replace microtime(true) with microtime_float().
> I uses ob_start and ob_end_flush, would that cause problems?
It increases the use of resources. When stretched it might be a problem,
normally it won't. I wouldn't think it's main cause of the problem, but it
will add to it. Is there a specific reason you NEED ob functions? If not:
don't use them.
To decrease server-load you could think about a cache system: review the
dependancies of databasefields for each of your pages, create appropriate
timestamps with an index in the database if they don't exist already, and
cache the pages locally as html with a date/time. On a request, check wether
there is a timestamp in the database higher than of your cached html If not,
serve the cached file, if so, create new cachefile and serve that. Creating
of cachefiles should offcourse take place automatically on an update of a
certain database field, but that might be even more work.
I hope I've been of some help,
--
Rik Wasmus
| |
|
| On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
<winandwaves@coldmail.com> wrote:
>Hi Folk
>
>I am managing a site, www.friars.co.nz that seems to be hitting the limits
>and I keep getting the 500 error. According to the people from webfarm it
>is because my script are too demanding or not closed properly.
>
>I dont believe you have to "close" PHP scripts or even database connections.
Ideally you shouldn't have to do either. PHP scripts automatically
close all resources upon termination, unless you've explicitly asked
them to do otherwise.
I've rarely encountered a 500 error using PHP, typically it crops up
with Perl scripts. In most situations, PHP will throw its own errors,
which are descriptive enough to trace the source of the problem.
(Granted, I just posted a few moments ago about an exception to this
case...)
>The site also accesses a 80 Megabyte database.
Database as in...? MySQL? How much traffic are you getting? Are you
using persistent database connections, or normal ones?
>Do you know of a way I can find out where the problems are or improving the
>performance of the site?
I'm not familiar with webfarm; with any luck, they give you a PHP
error log, or they send PHP's errors into Apache's error log. Whatever
error logs are available to you, check them religiously.
>I uses ob_start and ob_end_flush, would that cause problems?
Jerry did that one justice in his response.
hth
-
Remove mypants to email.
<http://www.shaunc.com/>
| |
| windandwaves 2006-04-29, 3:58 am |
| Jerry Stuckle wrote:
> windandwaves wrote:
>
> Well, ob_start and ob_end_flush require system resources; the bigger
> the page the more resources it requires.
>
> You might not be using many resources with it. You may be using a
> lot. It all depends on the site and what you're using it for.
Hmm, yes, the good thin with ob_end is that I can resize the actual HMTL to
be very small, meaning that what we send is small - at least.
| |
| windandwaves 2006-04-29, 3:58 am |
| Rik wrote:
....
> To decrease server-load you could think about a cache system: review
> the dependancies of databasefields for each of your pages, create
> appropriate timestamps with an index in the database if they don't
> exist already, and cache the pages locally as html with a date/time.
> On a request, check wether there is a timestamp in the database
> higher than of your cached html If not, serve the cached file, if so,
> create new cachefile and serve that. Creating of cachefiles should
> offcourse take place automatically on an update of a certain database
> field, but that might be even more work.
Rik, that is a  idea, would you just place the html in a database or is
there a special chache where pages are kept?
| |
| windandwaves 2006-04-29, 3:58 am |
| Shaun wrote:
> On Sat, 29 Apr 2006 13:40:42 +1200, "windandwaves"
> <winandwaves@coldmail.com> wrote:
>
Hmmm, yes, we have only about 10,000 unique visitors per month and a
relatively straight forward MySql database. However, we have shared
hosting, I think that is where the problem lies....
Thanks for your reply
[color=darkred]
> Nicolaas
| |
|
| windandwaves wrote:
> Rik wrote:
> ...
>
>
> Rik, that is a idea, would you just place the html in a database
> or is there a special chache where pages are kept?
An HTML page can perfectly be kept in the database, makes it even easier to
check wether a page has to be "rewritten" or not, possibly in 1 simple query
returning 0 or 1.
And if some items in a webpage aren't cachable, you could always try to
cache certain portions of code.
Grtz,
--
Rik Wasmus
| |
| Jerry Stuckle 2006-04-29, 6:58 pm |
| windandwaves wrote:
> Jerry Stuckle wrote:
>
>
>
> Hmm, yes, the good thin with ob_end is that I can resize the actual HMTL to
> be very small, meaning that what we send is small - at least.
>
>
Why even us it? I find very seldom do I need it. And it does add overhead.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
| |
| Jerry Stuckle 2006-04-29, 6:58 pm |
| windandwaves wrote:
> Shaun wrote:
>
>
>
>
> Hmmm, yes, we have only about 10,000 unique visitors per month and a
> relatively straight forward MySql database. However, we have shared
> hosting, I think that is where the problem lies....
>
> Thanks for your reply
>
>
>
>
>
10K visitors a month and an 80Mb database are nothing. Any shared host should
be able to handle that, also.
If you think it's a problem, talk to your hosting company. Maybe the server
you're on is overloaded and they will move you to a less heavily loaded server.
I'm not familiar with webfarm - but they may be right, also. For instance, if
you don't close your mysql connection, it will be closed eventually. But the
connection will hang around until the garbage collector gets around to cleaning
things up.
And look at how efficient your code is. I've seen some pretty good code - but
I've seen some very inefficient code out there, also. And the inefficient code
will require a lot more resources.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
| |
| windandwaves 2006-04-29, 6:58 pm |
| Jerry Stuckle wrote:
.....
> Why even us it? I find very seldom do I need it. And it does add
> overhead.
I use it on all my sites because in that way, I can perfectly indent my code
but take out the 100s or even 1000s of spaces/tabs.
For all my sites, I make a login and when the person is logged in (i.e. me
the administrator), the page is produced with spaces so that I can analyse
the code.
I like writing xhtml strict with real simple xhtml that is compressed (i.e.
without spaces) so that the actual pages are super small. I do this to
protest against the dreamweavers out there, who create html that is often
over 100kb and just pollutes the superhighway with endless <td><img
src="spacer.gif">etc......
HTMS
> Nicolaas
| |
| windandwaves 2006-04-29, 6:58 pm |
| Jerry Stuckle wrote:
> windandwaves wrote:
>
> 10K visitors a month and an 80Mb database are nothing. Any shared
> host should be able to handle that, also.
>
> If you think it's a problem, talk to your hosting company. Maybe the
> server you're on is overloaded and they will move you to a less
> heavily loaded server.
> I'm not familiar with webfarm - but they may be right, also. For
> instance, if you don't close your mysql connection, it will be closed
> eventually. But the connection will hang around until the garbage
> collector gets around to cleaning things up.
I will do a close database connection at the end of each script and see if
it makes a difference.
> And look at how efficient your code is. I've seen some pretty good
> code - but I've seen some very inefficient code out there, also. And
> the inefficient code will require a lot more resources.
It is so hard for me to judge if the code is efficient. Really - who knows.
I know what efficient code is, but there are just so many variables. Even
making changes tot the php.ini could be huge.
Each page loads about 100Kb of libaries and functions... (probably 50 - 100
functions (small ones) in total)... Does that make a difference?
I have a couple of tables in the database with over 300,000 (small) rows)...
Does that make a difference?
I use a lot of session variables (i.e. I keep a log of each page visited for
each person), but all of them are stored in highly optimised Mysql tables
(e.g.
person ID, page ID, time
person ID, page ID, time
person ID, page ID, time
person ID, page ID, time
In the end, because it is such muddy water, it is probably cheaper to pay
for a dedicated server than to spend hours optimising your code. I can do
fast development and write code in such a way that it is easy to maintain.
Thanks for your reply. Much appreciated.
> Nicolaas
| |
| Jerry Stuckle 2006-04-29, 9:58 pm |
| windandwaves wrote:
> Jerry Stuckle wrote:
> ....
>
>
>
> I use it on all my sites because in that way, I can perfectly indent my code
> but take out the 100s or even 1000s of spaces/tabs.
>
> For all my sites, I make a login and when the person is logged in (i.e. me
> the administrator), the page is produced with spaces so that I can analyse
> the code.
>
> I like writing xhtml strict with real simple xhtml that is compressed (i.e.
> without spaces) so that the actual pages are super small. I do this to
> protest against the dreamweavers out there, who create html that is often
> over 100kb and just pollutes the superhighway with endless <td><img
> src="spacer.gif">etc......
>
> HTMS
>
>
>
>
>
You can write perfectly good xml or html without obstart(). The two have
nothing to do with each other.
But compressing a page then expanding it just to make the code more readable
doesn't make sense. Just expand it in the file and serve it statically.
There's much less overhead.
I think the only time I've really needed obstart() is when I wanted to wrap
phpinfo() in another page. In that case I needed to get the output, parse it,
getting rid of the extra tags, then print it.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
| |
|
| windandwaves wrote:
> It is so hard for me to judge if the code is efficient. Really - who
> knows. I know what efficient code is, but there are just so many
> variables. Even making changes tot the php.ini could be huge.
Yup, but previous suggestion with microtime() would give you an indication
where the most time is taken.
> Each page loads about 100Kb of libaries and functions... (probably 50
> - 100 functions (small ones) in total)... Does that make a difference?
Yes, probably not a lot, but why?
I'm having great difficulties imagining a a website where that much
libraries and functions are needed. Maybe it's time to include only files
that are actually needed?
It may be worth it to check the following url:
http://nl2.php.net/manual/en/language.oop5.autoload.php
> I have a couple of tables in the database with over 300,000 (small)
> rows)... Does that make a difference?
Having a database doesn't make a differnce.
Exactly HOW you query the database does.
Normalize and create proper indexes.-
Grtz,
--
Rik Wasmus
| |
| Jerry Stuckle 2006-04-29, 9:58 pm |
| windandwaves wrote:
> Jerry Stuckle wrote:
>
>
>
> I will do a close database connection at the end of each script and see if
> it makes a difference.
>
>
>
>
> It is so hard for me to judge if the code is efficient. Really - who knows.
> I know what efficient code is, but there are just so many variables. Even
> making changes tot the php.ini could be huge.
>
> Each page loads about 100Kb of libaries and functions... (probably 50 - 100
> functions (small ones) in total)... Does that make a difference?
> I have a couple of tables in the database with over 300,000 (small) rows)...
> Does that make a difference?
> I use a lot of session variables (i.e. I keep a log of each page visited for
> each person), but all of them are stored in highly optimised Mysql tables
> (e.g.
>
> person ID, page ID, time
> person ID, page ID, time
> person ID, page ID, time
> person ID, page ID, time
>
> In the end, because it is such muddy water, it is probably cheaper to pay
> for a dedicated server than to spend hours optimising your code. I can do
> fast development and write code in such a way that it is easy to maintain.
>
> Thanks for your reply. Much appreciated.
>
>
>
>
>
>
Well, if every page is loading unnecessary functions, then yes, that will add
overhead. PHP has to parse all that code. I tend to break my functions in to
groups, and only load those which are necessary for the page.
The database itself isn't a problem. However, your access to the database could
be inefficient. For instance, you may be doing full table scans when the
appropriate index could allow an index scan. But that is all very dependent on
your the RDB you're using, database layout, the actual operations you're
performing on it and about a dozen other variables.
I wouldn't think session variables would have a huge effect on it, but it's also
possible. However, I would also look at what you're doing. Can you get
basically the same information from your apache log?
Yes, it will take time to make it more efficient. But I think it would be worth
it. And a dedicated server has its own additional overhead for maintenance.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
| |
| windandwaves 2006-04-29, 9:58 pm |
| Jerry Stuckle wrote:
> windandwaves wrote:
>
> You can write perfectly good xml or html without obstart(). The two
> have nothing to do with each other.
>
> But compressing a page then expanding it just to make the code more
> readable doesn't make sense. Just expand it in the file and serve it
> statically. There's much less overhead.
What I do is for any user that is not logged-in, I compress the page getting
rid of the 1000s of spaces and tabs. For the select few that log-in, I do
not compress it so I can quickly spot errors in the code.
> I think the only time I've really needed obstart() is when I wanted
> to wrap phpinfo() in another page. In that case I needed to get the
> output, parse it, getting rid of the extra tags, then print it.
I just like ditching all the html at once. Bang, 3Kb of perfectly clean
(x)html. The css, javascript and images are bulkier, but these are often
cached. Hmmm, you are probably right though, I would involved a LOT of
changes :-)
| |
| windandwaves 2006-04-29, 9:58 pm |
| Rik wrote:
> windandwaves wrote:
>
> Yup, but previous suggestion with microtime() would give you an
> indication where the most time is taken.
Yes, I may try that, but i found that pages load superfast one second and
almost not at all the next. I did some database work last night with a
script running through an entire table of 20,000 records one by one. I did
an echo for each line and it was really interesting to see that sometimes
the entire screen filled with echos within a split second, while the next
moment, each line would literally take one second.
>
>
> Yes, probably not a lot, but why?
> I'm having great difficulties imagining a a website where that much
> libraries and functions are needed. Maybe it's time to include only
> files that are actually needed?
Yes, that is for sure, but sorting out exactly what is needed for each page
take expensive development time.
>
> It may be worth it to check the following url:
> http://nl2.php.net/manual/en/language.oop5.autoload.php
>
>
> Having a database doesn't make a differnce.
> Exactly HOW you query the database does.
Lol, yes, I guess you are right. I have added lots of indexes so querying
should be pretty fast, but adding may take longer....
I have one unique index that compromises five fields, would that matter?
So many questions, so few clues
> Normalize and create proper indexes.-
>
> Grtz,
| |
| Jerry Stuckle 2006-04-29, 9:58 pm |
| windandwaves wrote:
> Jerry Stuckle wrote:
>
>
>
> What I do is for any user that is not logged-in, I compress the page getting
> rid of the 1000s of spaces and tabs. For the select few that log-in, I do
> not compress it so I can quickly spot errors in the code.
>
>
>
>
> I just like ditching all the html at once. Bang, 3Kb of perfectly clean
> (x)html. The css, javascript and images are bulkier, but these are often
> cached. Hmmm, you are probably right though, I would involved a LOT of
> changes :-)
>
>
And what happens if you have a problem with your compression?
I'm now beginning to see why your provider is suspecting your scripts are a
problem.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
| |
| windandwaves 2006-04-30, 8:00 am |
| Jerry Stuckle wrote:
....snip....
> And what happens if you have a problem with your compression?
>
> I'm now beginning to see why your provider is suspecting your scripts
> are a problem.
when I say compression, I just mean taking out the extra spaces, nothing
else. that is all I do. I load all the script outputs to the buffer and
then remove any double spaces and tabs. The funny thing is that in the last
day, the server runs a lot, lot faster.... With the same amount of visitors.
I am still
| |
| Jeff North 2006-04-30, 8:00 am |
| On Sun, 30 Apr 2006 13:44:43 +1200, in comp.lang.php "windandwaves"
<winandwaves@coldmail.com>
<uEU4g.6659$0Y5.5349@news.xtra.co.nz> wrote:
>| Rik wrote:
>| > windandwaves wrote:
>| >> It is so hard for me to judge if the code is efficient. Really - who
>| >> knows. I know what efficient code is, but there are just so many
>| >> variables. Even making changes tot the php.ini could be huge.
>| >
>| > Yup, but previous suggestion with microtime() would give you an
>| > indication where the most time is taken.
>|
>| Yes, I may try that, but i found that pages load superfast one second and
>| almost not at all the next. I did some database work last night with a
>| script running through an entire table of 20,000 records one by one. I did
>| an echo for each line and it was really interesting to see that sometimes
>| the entire screen filled with echos within a split second, while the next
>| moment, each line would literally take one second.
When was the last time you 'defragged' your tables?
Optimize table <tablename>
Repair table <tablename> EXTENDED USE_FRM
---------------------------------------------------------------
I often wish that email had never been invented, but there’s
just no way I can get rid of it. So, day after day, several times
a day, I dutifully delete 99% of the emails I receive, and when
I’m not able to get at my email for a few days, I’ll leave the
machine at home running to pick it up every 10 minutes so I don’t
overflow some capacity somewhere, and just the other day I caught
myself wondering who will clean out my Inbox after I’m dead.
Charles Petzold. October 20, 2005
---------------------------------------------------------------
|
|
|
|
|