Mercurial > hg > lukkari

annotate update.sh @ 151:ca012374190c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix update script to work with dash/non-bash shells.
author Matti Hamalainen <ccr@tnsp.org>
date Thu, 25 Sep 2014 19:27:34 +0300
parents 6f7a2f9dcad4
children 4085ea7aa7a6
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
1 #!/bin/sh
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
2 URLSUFFIX=".htm"
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
3
151
ca012374190c Fix update script to work with dash/non-bash shells.
Matti Hamalainen <ccr@tnsp.org>
parents: 134
diff changeset
4 createdir()
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
5 {
125
22872b46eee9 Cleanup.
Matti Hamalainen <ccr@tnsp.org>
parents: 124
diff changeset
6 test ! -d "$1" && mkdir -p "$1" && chmod 751 "$1"
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
7 }
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
8
125
22872b46eee9 Cleanup.
Matti Hamalainen <ccr@tnsp.org>
parents: 124
diff changeset
9
151
ca012374190c Fix update script to work with dash/non-bash shells.
Matti Hamalainen <ccr@tnsp.org>
parents: 134
diff changeset
10 fetch()
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
11 {
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
12 URLPREFIX="$1"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
13 CLASSFILE="$2"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
14 LISTFILE="$2.tmp"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
15 PATPREFIX="$3"
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
16
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
17 if wget -q -O "$LISTFILE" "$URLPREFIX"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
18 perl -ne "if (/<a href=\"${PATPREFIX}([A-Z]{3}\d\S+)${URLSUFFIX}\">/) { print \"\$1\n\"; }" < "$LISTFILE" > "$CLASSFILE"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
19 echo -n "* Fetched classfile $CLASSFILE: "
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
20 cat "$CLASSFILE" | wc -l
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
21 fi
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
22 }
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
23
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
24
151
ca012374190c Fix update script to work with dash/non-bash shells.
Matti Hamalainen <ccr@tnsp.org>
parents: 134
diff changeset
25 parse()
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
26 {
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
27 # Create cache directories, if they do not exist
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
28 OLDCACHEDIR="${CACHEDIR}old/"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
29 createdir "${OLDCACHEDIR}"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
30 createdir "${CACHEDIR}"
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
31
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
32 URLPREFIX="$1"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
33 CLASSFILE="$2"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
34 PATPREFIX="$3"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
35
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
36 echo "<? \$origBaseURI = \"${URLPREFIX}${PATPREFIX}\"; \$origBaseExt = \".htm\"; ?>" > "${CACHEDIR}baseuri.data";
82
c553ad61e9c2 Make original data links work for both current and next period mode.
Matti Hamalainen <ccr@tnsp.org>
parents: 76
diff changeset
37
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
38 test -e "$CLASSFILE" || return 0
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
39
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
40 echo "* Processing $CLASSFILE ..."
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
41 cat "$CLASSFILE" | while read i; do
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
42 parse=no
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
43 INFILE="${CACHEDIR}${i}.html"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
44 ONFILE="${OLDCACHEDIR}${i}.html"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
45 DATAFILE="${i}.data"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
46 wget -q -O "${INFILE}.new" "${URLPREFIX}${PATPREFIX}${i}${URLSUFFIX}"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
47
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
48 if test -e "${INFILE}.new"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
49 # New data fetched, does old file exist?
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
50 if test -e "$INFILE"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
51 # Yes, do a diff
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
52 if ! diff -u "$INFILE" "$INFILE.new" > "$INFILE.diff"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
53 # There were differences, do a parse
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
54 parse=yes
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
55 mv "$INFILE" "$ONFILE" && \
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
56 mv "$INFILE.new" "$INFILE"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
57 else
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
58 # No changes, apparently .. remove the new one
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
59 rm -f "$INFILE.new" "$INFILE.diff"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
60 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
61 else
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
62 # No old file, parse new data
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
63 mv "$INFILE.new" "$INFILE"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
64 parse=yes
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
65 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
66 fi
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
67
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
68 # No new file fetched, does datafile exist?
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
69 if test ! -e "${CACHEDIR}${DATAFILE}"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
70 # No, try to parse it if old file input exists
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
71 parse=yes
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
72 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
73
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
74 # Parsing of old data requested?
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
75 if test "x$parse" = "xyes" -a -e "$INFILE"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
76 if test -e "${CACHEDIR}${DATAFILE}"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
77 mv "${CACHEDIR}${DATAFILE}" "${OLDCACHEDIR}${DATAFILE}"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
78 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
79 echo "Parsing $i"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
80 perl parsedata.pl -php "$INFILE" -o "${CACHEDIR}${DATAFILE}"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
81 #perl parsedata.pl -xml "$INFILE" -o "${CACHEDIR}${i}.xml"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
82 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
83 done
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
84 }
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
85
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
86 # Fix permissions
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
87 chmod 751 . && chmod 644 *.php *.css && chmod 600 coursecache.txt classes.txt classes_next.txt
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
88
35
4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web.
Matti Hamalainen <ccr@tnsp.org>
parents: 34
diff changeset
89 CACHEDIR="cache/"
134
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
90 fetch "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
91 parse "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
92 #parse "http://www.oamk.fi/tyojarjestykset/otek/luokat/" "classes.txt" "OR_"
29
ac51fc10414f Add support for URL prefix pattern in updates.
Matti Hamalainen <ccr@tnsp.org>
parents: 24
diff changeset
93
122
8cbd07999b66 Update to latest upcoming.
Matti Hamalainen <ccr@tnsp.org>
parents: 116
diff changeset
94
31
dbe7ff545293 Add support for fetching and showing data for next/upcoming period.
Matti Hamalainen <ccr@tnsp.org>
parents: 29
diff changeset
95 CACHEDIR="cache-next/"
134
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
96 #fetch "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
97 #parse "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
102
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
98
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
99
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
100 # http://www.oamk.fi/~heikkim/riihi[1-5]/
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
101 # http://www.oamk.fi/~heikkim/Luhti[1-5]/