lukkari: update.sh annotate

annotate update.sh @ 35:4d9354abda73

Update fetching URLs and parameters to match the latest changes on OAMK's web.

author	Matti Hamalainen <ccr@tnsp.org>
date	Wed, 09 Nov 2011 22:23:25 +0200
parents	1f6be8457912
children	808cf4c768c4

rev	line source
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	1 #!/bin/sh
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	2 URLSUFFIX=".htm"
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	3
24 1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	4 function parse()
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	5 {
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	6 URLPREFIX="$1"
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	7 CLASSFILE="$2"
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	8 LISTFILE="$2.tmp"
29 ac51fc10414f Add support for URL prefix pattern in updates. Matti Hamalainen <ccr@tnsp.org> parents: 24 diff changeset	9 PATPREFIX="$3"
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	10
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	11 if wget -q -O "$LISTFILE" "$URLPREFIX"; then
29 ac51fc10414f Add support for URL prefix pattern in updates. Matti Hamalainen <ccr@tnsp.org> parents: 24 diff changeset	12 perl -ne "if (/<a href=\"${PATPREFIX}([A-Z]{3}\d\S+)${URLSUFFIX}\">/) { print \"\$1\n\"; }" < "$LISTFILE" > "$CLASSFILE"
34 1f6be8457912 Script output cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 31 diff changeset	13 echo -n "* Fetched classfile $CLASSFILE: "
1f6be8457912 Script output cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 31 diff changeset	14 cat "$CLASSFILE" \| wc -l
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	15 fi
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	16
24 1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	17 if test -e "$CLASSFILE"; then
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	18 cat "$CLASSFILE" \| while read i; do
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	19 parse=no
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	20 INFILE="${CACHEDIR}${i}.html"
29 ac51fc10414f Add support for URL prefix pattern in updates. Matti Hamalainen <ccr@tnsp.org> parents: 24 diff changeset	21 wget -q -O "$INFILE.new" "${URLPREFIX}${PATPREFIX}${i}${URLSUFFIX}"
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	22
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	23 if test -e "$INFILE.new"; then
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	24 # New data fetched, does old file exist?
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	25 if test -e "$INFILE"; then
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	26 # Yes, do a diff
22 f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	27 if ! diff -u "$INFILE" "$INFILE.new" > "$INFILE.diff"; then
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	28 # There were differences, do a parse
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	29 parse=yes
22 f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	30 mv "$INFILE" "$INFILE.old"
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	31 mv "$INFILE.new" "$INFILE"
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	32 fi
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	33 else
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	34 # No old file, parse new data
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	35 mv "$INFILE.new" "$INFILE"
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	36 parse=yes
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	37 fi
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	38 else
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	39 # No new file fetched, does datafile exist?
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	40 if test ! -e "$i.data"; then
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	41 # No, try to parse it if old file input exists
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	42 parse=yes
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	43 fi
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	44 fi
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	45
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	46 # Parsing of old data requested?
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	47 if test "x$parse" = "xyes" -a -e "$INFILE"; then
22 f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	48 OUTFILE="${CACHEDIR}/$i.data"
f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	49 if test -e "$OUTFILE"; then
f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	50 mv "$OUTFILE" "$OUTFILE.old"
f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	51 fi
34 1f6be8457912 Script output cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 31 diff changeset	52 echo "Parsing $i"
22 f025d36926a1 Make backup copies of old versions of data files when updating. Matti Hamalainen <ccr@tnsp.org> parents: 21 diff changeset	53 perl parsedata.pl -php "$INFILE" -o "$OUTFILE"
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	54 fi
7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	55 done
24 1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	56 fi
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also. Matti Hamalainen <ccr@tnsp.org> parents: 22 diff changeset	57 }
6 7fca87c41e17 Added data fetching and updating shellscript. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	58
35 4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web. Matti Hamalainen <ccr@tnsp.org> parents: 34 diff changeset	59 CACHEDIR="cache/"
4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web. Matti Hamalainen <ccr@tnsp.org> parents: 34 diff changeset	60 parse "http://www.oamk.fi/tyojarjestykset/otek/luokat/" "luokat.txt" "OR_"
4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web. Matti Hamalainen <ccr@tnsp.org> parents: 34 diff changeset	61 #parse "http://www.oamk.fi/~heikkim/Luhti1/Ryhm%84t/" "luokat.txt" "Ryh._"
29 ac51fc10414f Add support for URL prefix pattern in updates. Matti Hamalainen <ccr@tnsp.org> parents: 24 diff changeset	62
31 dbe7ff545293 Add support for fetching and showing data for next/upcoming period. Matti Hamalainen <ccr@tnsp.org> parents: 29 diff changeset	63 CACHEDIR="cache-next/"
35 4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web. Matti Hamalainen <ccr@tnsp.org> parents: 34 diff changeset	64 parse "http://www.oamk.fi/~heikkim/Luhti3/Ryhm%84t/" "luokat_next.txt" "Ryh._"

Mercurial > hg > lukkari

annotate update.sh @ 35:4d9354abda73