MAMEWorld >> Programming
View all threads Index   Threaded Mode Threaded  

Pages: 1

italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


PHP question, FTP and dir listings of larger sizes.
#87274 - 09/13/06 04:33 AM


I'm ftp'ing into a server, and if I nlist the root dir consisting of a few files, I can do it just fine.

When I change dirs and nlist the folder with 8,000 files in it, no go. Any ideas? Timeout of some kind? I figure I should be getting some kind of data off it....



twistyAdministrator
Space Lord
Reged: 09/18/03
Posts: 15570
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: italie]
#87294 - 09/13/06 06:16 AM


> I'm ftp'ing into a server, and if I nlist the root dir consisting of a few files, I
> can do it just fine.
>
> When I change dirs and nlist the folder with 8,000 files in it, no go. Any ideas?
> Timeout of some kind? I figure I should be getting some kind of data off it....

Try changing to passive mode first.


Code:

$ftp_host = "yourFTPHost";
$ftp_user = "yourUsername";
$ftp_password = "yourPassword";
$conn = ftp_connect($ftp_host);
ftp_login($conn, $ftp_user, $ftp_password);
ftp_pasv($conn, TRUE);
$file_list = ftp_nlist($conn, "");
foreach ($file_list as $file)
{
echo "<br>$file";
}
ftp_close($conn);







italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: twisty]
#87297 - 09/13/06 06:52 AM



> Try changing to passive mode first.
>
> $ftp_host = "yourFTPHost";
> $ftp_user = "yourUsername";
> $ftp_password = "yourPassword";
> $conn = ftp_connect($ftp_host);
> ftp_login($conn, $ftp_user, $ftp_password);
> ftp_pasv($conn, TRUE);
> $file_list = ftp_nlist($conn, "");
> foreach ($file_list as $file)
> {
> echo "
$file";
> }
> ftp_close($conn);


Just tried that a little bit ago, no dice. Might be 'one' of the issues so I've left it in my code.

It definatly seems to be a directory size issue. This code will work on any other directory I search, sans the big one. Can't seem to find any documentation about a size issue though.



twistyAdministrator
Space Lord
Reged: 09/18/03
Posts: 15570
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: italie]
#87300 - 09/13/06 07:06 AM


Just for fun, try adding this to the top of your script. Perhaps it's your end which is timing out on the lengthy retrieval.

set_time_limit(0);
ini_set("max_execution_time",0);






italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: twisty]
#87310 - 09/13/06 08:41 AM


> Just for fun, try adding this to the top of your script. Perhaps it's your end which
> is timing out on the lengthy retrieval.
>
> set_time_limit(0);
> ini_set("max_execution_time",0);

Nah, still choking....BUT....I've just noticed that it might have something to do with IE7.

IE7 affecting server side code....NICE.....



PiModerator
Allergic to life
Reged: 09/20/03
Posts: 6449
Loc: Room 101
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: italie]
#87473 - 09/14/06 04:50 PM


> I'm ftp'ing into a server, and if I nlist the root dir consisting of a few files, I
> can do it just fine.
>
> When I change dirs and nlist the folder with 8,000 files in it, no go. Any ideas?
> Timeout of some kind? I figure I should be getting some kind of data off it....

But what do you mean with "no go"? Does it stall? Does the result for ftp_nlist contain FALSE? (Use strict comparison for this last one). Is it a problem with your code or with the FTP? What PHP version are you using? Can you replicate the problem with your own local FTP server in your computer? Am I making too many questions ?



Wound up, can't sleep, can't do anything right, little honey / Oh, since I set my eyes on you. / I tell you the truth.
I can't get it right / Get it right / Since I met you...



twistyAdministrator
Space Lord
Reged: 09/18/03
Posts: 15570
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: Pi]
#87507 - 09/14/06 11:01 PM


> But what do you mean with "no go"? Does it stall? Does the result for ftp_nlist
> contain FALSE? (Use strict comparison for this last one). Is it a problem with your
> code or with the FTP? What PHP version are you using? Can you replicate the problem
> with your own local FTP server in your computer? Am I making too many questions ?

Yeah the exact error output using...

error_reporting(E_ALL);

...might help.






italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: twisty]
#87576 - 09/15/06 05:00 AM


>> But what do you mean with "no go"?

Sometimes it sits there at a white page, sometimes it gives me a "internet explorer cannot display this web page" page.

>>Does it stall?

Sometimes

>>Does the result for ftp_nlist contain FALSE? (Use strict comparison for this last one).

Can't say, still haven't gotten it to output ANYTHING when runing on that dir.

>>Is it a problem with your code or with the FTP?

Code works fine with smaller directories. Still could be a coding or ftp error, but it DOES work on smaller dirs.


>>What PHP version are you using?

on my host, 4.4.2

>>Can you replicate the problem with your own local FTP server in your computer?

Kind of a tricky question to answer. I'm running the php from a private host, but the FTP IS on my own computer right now. Will be untill I finish this coding.

The even bigger kicker is that the ftp server is on my server, and the folders I access are on a networked computer to the server via virtual folders. The chain of potential fuck up points is endless, but then again neither of my machines are throwing errors in the logs, and it DOES work with smaller folders.

>>Am I making too many questions ?

No, keep em coming. Helps me keep perspective.

>
> Yeah the exact error output using...
>
> error_reporting(E_ALL);
> ...might help.

Noting. Nada. Zip.

Gotta be a time out somewhere, or nlist having a shitfit at certain array sizes.



twistyAdministrator
Space Lord
Reged: 09/18/03
Posts: 15570
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: italie]
#87584 - 09/15/06 06:15 AM


> > error_reporting(E_ALL);
> > ...might help.
>
> Noting. Nada. Zip.
>
> Gotta be a time out somewhere, or nlist having a shitfit at certain array sizes.

Sounds like a config problem with the remote server then. Could be any number of things like you say. Perhaps a memory limit is being hit for such a large array.






PiModerator
Allergic to life
Reged: 09/20/03
Posts: 6449
Loc: Room 101
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: italie]
#87665 - 09/15/06 04:15 PM


> Gotta be a time out somewhere, or nlist having a shitfit at certain array sizes.

Judging by my experience, arrays in PHP can be *awfully* slow when they are really big, but there's no reason for a complete stall.

About not outputting anything, have you tried writing to text files?

Another thing to consider is how are you developing PHP in a remote machine. I thought you'd have a local AMP (Apache MySQL PHP) server and you were doing the work there. A debugger can be easily placed somewhere in there, and trust me, debugging PHP is about 95% of the work of fixing code. What I have here is a copy *cough* of NuSphere phpED, which comes with a debugger you can run under PHP (as a module) and helps so much that it's almost pornographic.

In any case, I'm thinking that the problem is with ftp_nlist which for some reason chokes in large directories, and that can't be debugged. But it would be a step ahead if you could be sure that the problem is within a command.

A final thought is discovering when does ftp_nlist stop working. Is it at 8000 files? 4000? A binary search shouldn't take really long. I would also compare times of execution depending on directory size.



Wound up, can't sleep, can't do anything right, little honey / Oh, since I set my eyes on you. / I tell you the truth.
I can't get it right / Get it right / Since I met you...



italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: Pi]
#87675 - 09/15/06 05:26 PM


> About not outputting anything, have you tried writing to text files?

not yet, but based upon the type of stall, I doubt it will write out to one.

> In any case, I'm thinking that the problem is with ftp_nlist which for some reason
> chokes in large directories, and that can't be debugged. But it would be a step ahead
> if you could be sure that the problem is within a command.

Working on doing that now, just trying to develop a solid plan to prove it. I've pretty much ruled out the online host running php. I'm 99% sure its a core bug with nlist.


> A final thought is discovering when does ftp_nlist stop working. Is it at 8000 files?
> 4000? A binary search shouldn't take really long. I would also compare times of
> execution depending on directory size.

That would require a LOT of file shuffling, but it might come down to that. I'll update when I've figured out more. Thanks for all the advice guys.



italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Almost there.... new [Re: italie]
#87922 - 09/17/06 11:21 PM


I had max_execution_time set to "0", which I assumed to be infinite. Changing that brought the script back to life for large dirs in firefox, but not in IE.

we'll see whats the hangup there.



PiModerator
Allergic to life
Reged: 09/20/03
Posts: 6449
Loc: Room 101
Send PM


Re: Almost there.... new [Re: italie]
#87926 - 09/18/06 01:13 AM


> I had max_execution_time set to "0", which I assumed to be infinite. Changing that
> brought the script back to life for large dirs in firefox, but not in IE.

Interesting. It might have to do with different rendering methods, which might cause (?) extraneously large times in IE. Try with ob_implicit_flush() and a smart flush() usage to see if it helps in the output and it shows something before stalling.

Does the script end if you don't try any large output?



Wound up, can't sleep, can't do anything right, little honey / Oh, since I set my eyes on you. / I tell you the truth.
I can't get it right / Get it right / Since I met you...



twistyAdministrator
Space Lord
Reged: 09/18/03
Posts: 15570
Send PM


Re: Almost there.... new [Re: italie]
#87930 - 09/18/06 01:35 AM


> I had max_execution_time set to "0", which I assumed to be infinite.

It definitely means that. How weird.






PiModerator
Allergic to life
Reged: 09/20/03
Posts: 6449
Loc: Room 101
Send PM


Re: Almost there.... new [Re: italie]
#87934 - 09/18/06 02:31 AM


> I had max_execution_time set to "0", which I assumed to be infinite.

Forgot to mention that you should use the function set_time_limit() instead of changing php.ini. 0 means "no limit".



italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Re: Almost there.... new [Re: Pi]
#87936 - 09/18/06 02:41 AM


> > I had max_execution_time set to "0", which I assumed to be infinite.
>
> Forgot to mention that you should use the function set_time_limit() instead of
> changing php.ini. 0 means "no limit".

already using that, but thanks for the info. Changed it to 900 just in case a similar issue is present.



italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Dead again.... new [Re: Pi]
#87937 - 09/18/06 02:49 AM


> > I had max_execution_time set to "0", which I assumed to be infinite. Changing that
> > brought the script back to life for large dirs in firefox, but not in IE.
>
> Interesting. It might have to do with different rendering methods, which might cause
> (?) extraneously large times in IE. Try with ob_implicit_flush() and a smart flush()
> usage to see if it helps in the output and it shows something before stalling.
>
> Does the script end if you don't try any large output?

working flawlessly running php from my test box in firefox, but dead when uploading to the host. At least I know where to start looking for that issue.

What do you mean by large output? Script runs through completely for smaller directories. Larger directories hang at a white browser page with no response to any code, and eventually time out. As I said, the script is working great now on my test box after changing the max_ex_time from zero to 120. Here is what I'm using code wise for a simplified test page, see anyhting out of the ordinary? I'll try flushing it when I can get it back up on firefox.




Code:

 



//$ftp_tout = 900; // timeout
ini_set("memory_limit","50M");
ini_set("set_time_limit", "120");
ini_set("max_execution_time","120");
error_reporting(E_ALL);

$ftp_server = '67.xx.xx.xx';
$ftp_user_name = 'xxx';
$ftp_user_pass = 'xxxx';




$conn_id = ftp_connect($ftp_server);


$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass);

//
//Enable PASV ( Note: must be done after ftp_login() )
//

$mode = ftp_pasv($conn_id, TRUE);



if ((!$conn_id) || (!$login_result) || (!$mode))
{
echo "FTP connection has failed!";
echo "Attempted to connect to $ftp_server for user $ftp_user_name";
exit;
}
else
{
echo "Connected to $ftp_server, for user $ftp_user_name connection ID is $conn_id

";
// change dir to music


$dir=ftp_pwd($conn_id);

echo "$dir
";

ftp_chdir($conn_id, "xxxxxx");

$dir=ftp_pwd($conn_id);
echo "$dir
";


//
//Now run ftp_nlist()
//

$file_list = ftp_nlist($conn_id, " ");
foreach ($file_list as $file)
{
echo "
$file";
}

echo"
Through for each loop";

ftp_close($conn_id);

%>







PiModerator
Allergic to life
Reged: 09/20/03
Posts: 6449
Loc: Room 101
Send PM


Re: Dead again.... new [Re: italie]
#88000 - 09/18/06 06:27 PM



Quote:



$file_list = ftp_nlist($conn_id, " ");
foreach ($file_list as $file)
{
echo "
$file";
}





If instead of the foreach loop, you simply output the number of elements in the array $file_list, does the script finish properly? In other words, are you sure that the script stalls at ftp_nlist or at the foreach loop?



italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


update.... new [Re: Pi]
#92337 - 11/05/06 07:02 PM


Finally got back to this one, seem to have the hang ups solved. You appear to have hit it on the head Pi, it was the loops. The foreach loop would run flawlessly on firefox, but IE7 had a shitfit when running foreach on large dirs over +5000 files.. Changing that particular loop to a plain vannila for loop ending at the array size works in all browsers tried. Strange that IE7 would freak over one valid line of php like that depending on what was thrown at it.


> $file_list = ftp_nlist($conn_id, " ");
> foreach ($file_list as $file)
> {
> echo "
> $file";
> }
>
> If instead of the foreach loop, you simply output the number of elements in the array
> $file_list, does the script finish properly? In other words, are you sure that the
> script stalls at ftp_nlist or at the foreach loop?



twistyAdministrator
Space Lord
Reged: 09/18/03
Posts: 15570
Send PM


Re: update.... new [Re: italie]
#92338 - 11/05/06 07:08 PM


> but IE7 had a shitfit

Are you noticing a pattern yet?



italieAdministrator
MAME owes italie many thank yous, hah
Reged: 09/20/03
Posts: 15246
Loc: BoomTown
Send PM


Re: update.... new [Re: twisty]
#92340 - 11/05/06 07:20 PM


> > but IE7 had a shitfit
>
> Are you noticing a pattern yet?

no, should I be?









in all honesty I could have done with ignoring the problem, but IE7 is about to be a rampant infestation on the net (mark my words), so I might as well start dealing with the issues now.



PiModerator
Allergic to life
Reged: 09/20/03
Posts: 6449
Loc: Room 101
Send PM


Re: update.... new [Re: italie]
#92708 - 11/09/06 01:28 AM


> Finally got back to this one, seem to have the hang ups solved. You appear to have
> hit it on the head Pi, it was the loops. The foreach loop would run flawlessly on
> firefox, but IE7 had a shitfit when running foreach on large dirs over +5000 files..
> Changing that particular loop to a plain vannila for loop ending at the array size
> works in all browsers tried. Strange that IE7 would freak over one valid line of php
> like that depending on what was thrown at it.

I am glad that you were able to fix it, but how can it be a browser problem when PHP is a server-side script language? In other words, IE7 doesn't run a single PHP line. It only outputs the html or text that the PHP spits out. So, maybe the way the output is being sent to the browser (with frequent pauses, irregular block sizes, etc.) makes IE7 freak out. Maybe IE7 has a streaming or buffering problem. And that would be a really BAD problem, that is a basic area to have a bug there...

I'm also glad that I was able to help, I was running out of possibilities. Fixing a bug is easy, finding why a supposedly correct code has a bug, and where exactly is the tricky part. When I started spending more time debugging than writing code, I started SANE coding habits to avoid those stupid and unfindable bugs.



kgutteridge
MAME Fan
Reged: 01/17/07
Posts: 3
Send PM


Re: PHP question, FTP and dir listings of larger sizes. new [Re: italie]
#99150 - 01/17/07 03:39 AM


sure you arent over running the php max execution time in the php.ini, the default is 30 or 60 seconds, if you pinging a shared hosting box this could very well be the case


Pages: 1

MAMEWorld >> Programming
View all threads Index   Threaded Mode Threaded  

Extra information Permissions
Moderator:  Pi 
0 registered and 29 anonymous users are browsing this forum.
You cannot start new topics
You cannot reply to topics
HTML is enabled
UBBCode is enabled
Thread views: 4808