Mercurial > hg > lukkari

annotate update.sh @ 193:16ce445c499a v3 tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Import v3 branch.
author Matti Hamalainen <ccr@tnsp.org>
date Sun, 22 Jan 2017 02:31:10 +0200
parents b4a07ea2d739
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
1 #!/bin/sh
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
2 URLSUFFIX=".htm"
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
3
151
ca012374190c Fix update script to work with dash/non-bash shells.
Matti Hamalainen <ccr@tnsp.org>
parents: 134
diff changeset
4 createdir()
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
5 {
125
22872b46eee9 Cleanup.
Matti Hamalainen <ccr@tnsp.org>
parents: 124
diff changeset
6 test ! -d "$1" && mkdir -p "$1" && chmod 751 "$1"
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
7 }
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
8
125
22872b46eee9 Cleanup.
Matti Hamalainen <ccr@tnsp.org>
parents: 124
diff changeset
9
151
ca012374190c Fix update script to work with dash/non-bash shells.
Matti Hamalainen <ccr@tnsp.org>
parents: 134
diff changeset
10 fetch()
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
11 {
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
12 URLPREFIX="$1"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
13 CLASSFILE="$2"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
14 LISTFILE="$2.tmp"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
15 PATPREFIX="$3"
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
16
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
17 if wget -q -O "$LISTFILE" "$URLPREFIX"; then
160
a84b40bc2a99 Add parselist.pl utility and use it in update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 157
diff changeset
18 perl -w ./parselist.pl "$PATPREFIX" "$URLSUFFIX" < "$LISTFILE" > "$CLASSFILE"
152
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
19 rm -f "$LISTFILE"
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
20 fi
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
21 }
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
22
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
23
151
ca012374190c Fix update script to work with dash/non-bash shells.
Matti Hamalainen <ccr@tnsp.org>
parents: 134
diff changeset
24 parse()
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
25 {
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
26 # Create cache directories, if they do not exist
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
27 OLDCACHEDIR="${CACHEDIR}old/"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
28 createdir "${OLDCACHEDIR}"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
29 createdir "${CACHEDIR}"
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
30
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
31 URLPREFIX="$1"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
32 CLASSFILE="$2"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
33 PATPREFIX="$3"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
34
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
35 echo "<? \$origBaseURI = \"${URLPREFIX}${PATPREFIX}\"; \$origBaseExt = \".htm\"; ?>" > "${CACHEDIR}baseuri.data";
82
c553ad61e9c2 Make original data links work for both current and next period mode.
Matti Hamalainen <ccr@tnsp.org>
parents: 76
diff changeset
36
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
37 test -e "$CLASSFILE" || return 0
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
38
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
39 echo "* Processing $CLASSFILE ..."
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
40 cat "$CLASSFILE" | while read i; do
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
41 parse=no
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
42 INFILE="${CACHEDIR}${i}.html"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
43 ONFILE="${OLDCACHEDIR}${i}.html"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
44 DATAFILE="${i}.data"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
45 wget -q -O "${INFILE}.new" "${URLPREFIX}${PATPREFIX}${i}${URLSUFFIX}"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
46
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
47 if test -e "${INFILE}.new"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
48 # New data fetched, does old file exist?
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
49 if test -e "$INFILE"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
50 # Yes, do a diff
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
51 if ! diff -u "$INFILE" "$INFILE.new" > "$INFILE.diff"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
52 # There were differences, do a parse
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
53 parse=yes
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
54 mv "$INFILE" "$ONFILE" && \
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
55 mv "$INFILE.new" "$INFILE"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
56 else
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
57 # No changes, apparently .. remove the new one
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
58 rm -f "$INFILE.new" "$INFILE.diff"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
59 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
60 else
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
61 # No old file, parse new data
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
62 mv "$INFILE.new" "$INFILE"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
63 parse=yes
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
64 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
65 fi
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
66
124
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
67 # No new file fetched, does datafile exist?
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
68 if test ! -e "${CACHEDIR}${DATAFILE}"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
69 # No, try to parse it if old file input exists
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
70 parse=yes
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
71 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
72
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
73 # Parsing of old data requested?
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
74 if test "x$parse" = "xyes" -a -e "$INFILE"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
75 if test -e "${CACHEDIR}${DATAFILE}"; then
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
76 mv "${CACHEDIR}${DATAFILE}" "${OLDCACHEDIR}${DATAFILE}"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
77 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
78 echo "Parsing $i"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
79 perl parsedata.pl -php "$INFILE" -o "${CACHEDIR}${DATAFILE}"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
80 #perl parsedata.pl -xml "$INFILE" -o "${CACHEDIR}${i}.xml"
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
81 fi
4547a239b0fd Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents: 122
diff changeset
82 done
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
83 }
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
84
152
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
85 fix_permissions()
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
86 {
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
87 chmod 751 . && chmod 644 *.php *.css && chmod 600 coursecache.txt classes.txt classes_next.txt
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
88 }
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
89
152
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
90 fix_permissions
35
4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web.
Matti Hamalainen <ccr@tnsp.org>
parents: 34
diff changeset
91 CACHEDIR="cache/"
134
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
92 fetch "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
93 parse "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
94 #parse "http://www.oamk.fi/tyojarjestykset/otek/luokat/" "classes.txt" "OR_"
29
ac51fc10414f Add support for URL prefix pattern in updates.
Matti Hamalainen <ccr@tnsp.org>
parents: 24
diff changeset
95
122
8cbd07999b66 Update to latest upcoming.
Matti Hamalainen <ccr@tnsp.org>
parents: 116
diff changeset
96
31
dbe7ff545293 Add support for fetching and showing data for next/upcoming period.
Matti Hamalainen <ccr@tnsp.org>
parents: 29
diff changeset
97 CACHEDIR="cache-next/"
134
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
98 #fetch "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
6f7a2f9dcad4 Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents: 125
diff changeset
99 #parse "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._"
102
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
100
152
4085ea7aa7a6 Few minor improvements to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 151
diff changeset
101 fix_permissions
157
e97705171c3c Show diff of class files.
Matti Hamalainen <ccr@tnsp.org>
parents: 152
diff changeset
102 diff -u "classes.txt" "classes_next.txt"
102
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
103
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
104 # http://www.oamk.fi/~heikkim/riihi[1-5]/
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
105 # http://www.oamk.fi/~heikkim/Luhti[1-5]/