Stiletto |
|
They're always after me Lucky ROMS!
|
|
|
|
|
|
Reged: 03/07/04
|
|
Posts: 6472
|
|
|
|
|
|
Send PM
|
|
|
Batch file processing...
#106614 - 03/29/07 08:12 AM
|
|
|
Okay, so on a work computer I have a file directory similar to this:
STATES
|_ ALABAMA
|_ ALASKA
.
.
.
|_ WYOMING
In each folder there are large numbers of "proprietary" binary files, with each file having large amounts of ASCII text inside. The file is named like so: DATA - CITY1.stl. In my example, this will be "Joe Smith - Detroit.stl"
The inside of a file would be similar to so:
...header...
"NAME"="Joe Smith"
"CITY2"="Detroit"
"PHONE"="3015551212"
"IMAGE"=encoded monochrome image
...other fields...
...footer...
I'd like a script/app that goes through the folder, including all subdirectories, extracts the DATA, CITY1, NAME, CITY2 and PHONE fields from each file, and concatenates it all into a CSV file where a sample CSV would look like so:
DATA,CITY1,NAME,CITY2,PHONE
Joe Smith,Detroit,Joe Smith,Detroit,3015551212
The character combination to parse the filename will always be " - ". That's right, whitespace-hyphen-whitespace.
grep should do the trick, as would a Perl script, but it's been a good decade since I've dealt with either.
Anyone kind enough to save me some time? Once I have the feel for it, I should be able to fill in the particular ASCII to hunt for, as far as file data goes, myself.
I _can_ do it myself, but it will just take longer that way. I have a feeling someone here should be able to code this without even realizing it.
For bonus points, insert another field in the CSV that displays "% similar" statistic in comparing CITY1 and CITY2 fields.
In return, you get a tasty pudding snack. 
- Stiletto
Edited by Stiletto (03/29/07 08:47 AM)
|
|
|
|
Re: Batch file processing...
[Re: Stiletto]
#106644 - 03/29/07 08:58 PM
|
|
|
> Okay, so on a work computer I have a file directory similar to this:
{SNIPPIY-SNIP}
I could probably write something like that in Java. I have similar file processing programs that I've written that I could just cannibalize.
Would this be a one-off program or something that you need to run daily?
Should your output CSV have the STATE as well? or is there going to be one CSV per state?
|
GroovyMAME support forum on BYOAC
|
|
| Stiletto |
|
They're always after me Lucky ROMS!
|
|
|
|
|
|
Reged: 03/07/04
|
|
Posts: 6472
|
|
|
|
|
|
Send PM
|
|
|
Re: Batch file processing...
[Re: krick]
#106670 - 03/30/07 11:00 AM
|
|
|
> > Okay, so on a work computer I have a file directory similar to this:
>
> {SNIPPIY-SNIP}
>
> I could probably write something like that in Java. I have similar file processing
> programs that I've written that I could just cannibalize.
Great, thankee kindly.
> Would this be a one-off program or something that you need to run daily?
Basically one-off, not something production-quality. I may have need to run it again eventually.
> Should your output CSV have the STATE as well? or is there going to be one CSV per
> state?
No, STATE is not important, although the complete filename with path would certainly not go amiss since the plan is to only have one CSV. Basically the goal is to partially automate comparing to see if CITY1 matches CITY2. If it's completely off, then usually it will mean that the content of the file had been overwritten but the filename remained unchanged. This however does not check for the case where NAME or PHONE no longer match CITY2, but the only way I have to check for that case is manually. On over 1000 files. Arrgh!
I was however wrong about the cutoff ASCII within the file. The file content is more like this:
...header...
NAME Joe Smith
CITY2 Detroit
PHONE 3015551212
IMAGE encoded monochrome image
...other fields...
...footer...
- Stiletto
|
|
|
|
Re: Batch file processing...
[Re: Stiletto]
#106706 - 03/30/07 09:08 PM
|
|
|
> > > Okay, so on a work computer I have a file directory similar to this:
> >
> > {SNIPPIY-SNIP}
> >
> > I could probably write something like that in Java. I have similar file processing
> > programs that I've written that I could just cannibalize.
>
> Great, thankee kindly.
>
> > Would this be a one-off program or something that you need to run daily?
>
> Basically one-off, not something production-quality. I may have need to run it again
> eventually.
>
> > Should your output CSV have the STATE as well? or is there going to be one CSV per
> > state?
>
> No, STATE is not important, although the complete filename with path would certainly
> not go amiss since the plan is to only have one CSV. Basically the goal is to
> partially automate comparing to see if CITY1 matches CITY2. If it's completely off,
> then usually it will mean that the content of the file had been overwritten but the
> filename remained unchanged. This however does not check for the case where NAME or
> PHONE no longer match CITY2, but the only way I have to check for that case is
> manually. On over 1000 files. Arrgh!
>
> I was however wrong about the cutoff ASCII within the file. The file content is more
> like this:
>
> ...header...
> NAME Joe Smith
> CITY2 Detroit
> PHONE 3015551212
> IMAGE encoded monochrome image
> ...other fields...
> ...footer...
>
> - Stiletto
Wait... are these files binary? All the tools I've written in java were text based "line at a time" processing tools. If they're binary files, it might take a bit of reworking. Do you have a sample file that I could experiment with? Are the headers fixed length?
|
GroovyMAME support forum on BYOAC
|
|
| Stiletto |
|
They're always after me Lucky ROMS!
|
|
|
|
|
|
Reged: 03/07/04
|
|
Posts: 6472
|
|
|
|
|
|
Send PM
|
|
|
Re: Batch file processing...
[Re: krick]
#106737 - 03/31/07 09:48 AM
|
|
|
> > > > Okay, so on a work computer I have a file directory similar to this:
> > >
> > > {SNIPPIY-SNIP}
> > >
> > > I could probably write something like that in Java. I have similar file
> processing
> > > programs that I've written that I could just cannibalize.
> >
> > Great, thankee kindly.
> >
> > > Would this be a one-off program or something that you need to run daily?
> >
> > Basically one-off, not something production-quality. I may have need to run it
> again
> > eventually.
> >
> > > Should your output CSV have the STATE as well? or is there going to be one CSV
> per
> > > state?
> >
> > No, STATE is not important, although the complete filename with path would
> certainly
> > not go amiss since the plan is to only have one CSV. Basically the goal is to
> > partially automate comparing to see if CITY1 matches CITY2. If it's completely off,
> > then usually it will mean that the content of the file had been overwritten but the
> > filename remained unchanged. This however does not check for the case where NAME or
> > PHONE no longer match CITY2, but the only way I have to check for that case is
> > manually. On over 1000 files. Arrgh!
> >
> > I was however wrong about the cutoff ASCII within the file. The file content is
> more
> > like this:
> >
> > ...header...
> > NAME Joe Smith
> > CITY2 Detroit
> > PHONE 3015551212
> > IMAGE encoded monochrome image
> > ...other fields...
> > ...footer...
> >
> > - Stiletto
>
> Wait... are these files binary? All the tools I've written in java were text based
> "line at a time" processing tools. If they're binary files, it might take a bit of
> reworking. Do you have a sample file that I could experiment with? Are the headers
> fixed length?
Pretty sure it's a text file, each new field for information gets interpreted as being on a new line by NOTEPAD, so there must be newline chars at the end of each line. But sure, I'll upload a file tomorrow.
- Stiletto
|
|
|
|
Re: Batch file processing...
[Re: Stiletto]
#106904 - 04/02/07 07:26 PM
|
|
|
Is this the phone book of your cell phone or something?
smf
|
|
|
| Stiletto |
|
They're always after me Lucky ROMS!
|
|
|
|
|
|
Reged: 03/07/04
|
|
Posts: 6472
|
|
|
|
|
|
Send PM
|
|
|
Re: Batch file processing...
[Re: ]
#106955 - 04/03/07 08:03 AM
|
|
|
> Is this the phone book of your cell phone or something?
No, I changed the data field names to hide what I'm working with. And it turned out I'm wrong, again.
File contents are like so:
...header...
Object PHONE
...
...
...
DataValue=3015551212
End
Object NAME
...
...
...
DataValue=Joe Smith
End
Object CITY2
...
...
...
DataValue=Detroit
End
...
...footer...
Why, are you having a laff at how much a n00b I am? As I've said before, if you don't use it, you lose it. I haven't written my own code in a good 8-9 years, and see where I'm at now.
Krick, check your PM's.
- Stiletto
|
|
|
|
Re: Batch file processing...
[Re: Stiletto]
#107319 - 04/06/07 02:17 PM
|
|
|
> Why, are you having a laff at how much a n00b I am? As I've said before, if you don't
> use it, you lose it. I haven't written my own code in a good 8-9 years, and see where
> I'm at now.
I was just interested as my phone outputs a similar format to what you posted.
smf
|
|
|
| Stiletto |
|
They're always after me Lucky ROMS!
|
|
|
|
|
|
Reged: 03/07/04
|
|
Posts: 6472
|
|
|
|
|
|
Send PM
|
|
|
Re: Batch file processing...
[Re: ]
#107380 - 04/07/07 07:56 AM
|
|
|
> > Why, are you having a laff at how much a n00b I am? As I've said before, if you
> don't
> > use it, you lose it. I haven't written my own code in a good 8-9 years, and see
> where
> > I'm at now.
>
> I was just interested as my phone outputs a similar format to what you posted.
Okay, then - sorry to take it personal.
To interested onlookers:
Kricky fixed me up with a Java app with a GUI (hell, all I was expecting was a Win32 command line app) and it worked correctly on the first try, although to be technical at this stage all it does is one folder at a time.
I have to fix a bunch of filenames to get the parsing right, but my data audit will go a lot more smoothly now.
Thanks again, Krick!!!
---------------
The next one I'll do on me own: a handy-dandy calculator to calculate something our business pays attention to, every day, on the spot. I should be fine with that, except I might want a hand converting it from Win32 command line to Win32 UI once I have it working.
After that, I should be good to go!
- Stiletto
|
|
|
|
Re: Batch file processing...
[Re: Stiletto]
#107390 - 04/07/07 10:18 AM
|
|
|
|
|
| Stiletto |
|
They're always after me Lucky ROMS!
|
|
|
|
|
|
Reged: 03/07/04
|
|
Posts: 6472
|
|
|
|
|
|
Send PM
|
|
|
Re: Batch file processing...
[Re: ]
#107765 - 04/11/07 09:45 AM
|
|
|
> > I should be fine with that,
> > except I might want a hand converting it from Win32 command line to Win32 UI once I
> > have it working.
>
> http://msdn.microsoft.com/vstudio/express/visualcsharp/
>
> smf
I have this hangup about running code through an interpreter/virtual machine thing like the CLI. I understand the idea is portable rapid prototyping, and rapid prototyping is basically what I'm doing, but still.
I guess I'll get over this hangup at some point, rumor has it that I won't have any choice...
And all the programmers here shall roll their eyes at the n00b.
- Stiletto
|
|
|
| R. Belmont |
|
Cuckoo for IGAvania
|
|
|
|
|
|
Reged: 09/21/03
|
|
Posts: 9716
|
|
Loc: ECV-197 The Orville
|
|
|
|
Send PM
|
|
|
Re: Batch file processing...
[Re: Stiletto]
#108300 - 04/16/07 05:48 PM
|
|
|
> I have this hangup about running code through an interpreter/virtual machine thing
> like the CLI. I understand the idea is portable rapid prototyping, and rapid
> prototyping is basically what I'm doing, but still.
I guess I don't know why - it's much easier than either of the major "traditional" schools of Win32 GUI programming (MFC and Petzold), and for most GUI apps most of the CPU time is spent in the OS components so you can afford to lose a little speed.
Heck, the newest edition of The Petzold Book covers C# and Windows.Forms. The function of the sample apps hasn't changed since about Windows 3.0, but the implementation sure has
Edited by R. Belmont (04/16/07 05:49 PM)
|
|
|
Index 
Threaded 
Batch file processing...
Reply
[Re: 
