Minor py improvements

Author: khemarato

_content/articles/notes-on-the-udana_ireland.md

@@ -3,7 +3,8 @@ title: "Notes on the Udana"
 authors:
   - ireland
 source_url: "https://journal.equinoxpub.com/BSR/article/view/15266/23244"
-file_links: "exclusive_01/Notes%20on%20the%20Udana%20-%20John%20Ireland.pdf"
+file_links:
+  - "exclusive_01/Notes%20on%20the%20Udana%20-%20John%20Ireland.pdf"
 drive_links:
   - "https://drive.google.com/file/d/1CaGZC8D6v-O8ra2yLb-4WbEQscEG1Arj/view?usp=drivesdk"
 course: ud

_data/content.yml

@@ -1,5 +1,9 @@
 ## File links
 filehost: "https://buddhistuniversity.net/"
+valid_buckets:
+  - "exclusive_01"
+  - "patanjali-yoga-sutra"
+  - "function-data"
 noreferrerhosts:
   - "https://www.shin-ibs.edu"
 

_tests/content.md

@@ -13,7 +13,7 @@ A series of tests checking the integrity of the _content data
 | External URLs are URLs | {% assign ac = site.content | where_exp: "c", "c.external_url" %}{% assign fails = "" | split: "" %}{% for c in ac %}{% unless c.external_url contains "http" %}{% assign fails = fails | push: c %}{% endunless %}{% endfor %}{% if fails.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %}  |  {% if fails.size > 0 %}Found these bad URLs: {% for c in fails %}[{{ c.title | split: ":" | first }}]({{ c.url }}) {% endfor %}{% else %}No URLs missing http.{% endif %} 
 | Drive Links are Unique | {% assign ac = site.content | where_exp: "c", "c.drive_links" %}{% assign seen = "" | split: "" %}{% assign dupes = "" | split: "" %}{% for c in ac %}{% for d in c.drive_links %}{% if seen contains d %}{% unless dupes contains d %}{% assign dupes = dupes | push: d %}{% endunless %}{% else %}{% assign seen = seen | push: d %}{% endif %}{% endfor %}{% endfor %}{% if dupes.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %}  |  {% if dupes.size > 0 %}Found these duplicate links: `{{ dupes | join: " " }}`{% else %}No duplicate links found.{% endif %}
 | Course in tags | {% assign failures = site.content | where_exp: "c", "c.tags contains c.course" %}{% if failures.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %}  | There are {{ failures.size }} pieces with the course in the tags{% if failures.size > 0 %}: {{ failures | map: "slug" | array_to_sentence_string }}{% endif %}.
-| Sane file_links | {% assign valid_buckets = "exclusive_01,patanjali-yoga-sutra,function-data" | split: "," %}{% assign hostedcontent = site.content | where_exp: "c", "c.file_links.size > 0" %}{% assign fs = '' | split: '' %}{% for c in hostedcontent %}{% for l in c.file_links %}{% assign s = l | split: "/" %}{% if valid_buckets contains s[0] %}{% if s contains "blob" %}{% assign fs = fs | push: c %}{% endif %}{% else %}{% assign fs = fs | push: c %}{% endif %}{% endfor %}{% endfor %}{% if fs.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %}  |  {% assign count = hostedcontent.size | minus: fs.size %}{% if count > 0 %}There are {{ count }} sane links{% endif %}{% if fs.size > 0 %}The {{ fs.size }} pieces with bad buckets are: {% for c in fs %}[{{ c.title | split: ":" | first }}]({{ c.url }}) {% endfor %}{% endif %}
+| Sane file_links | {% assign hostedcontent = site.content | where_exp: "c", "c.file_links.size > 0" %}{% assign fs = '' | split: '' %}{% for c in hostedcontent %}{% for l in c.file_links %}{% assign s = l | split: "/" %}{% if site.data.content.valid_buckets contains s[0] %}{% if s contains "blob" %}{% assign fs = fs | push: c %}{% endif %}{% else %}{% assign fs = fs | push: c %}{% endif %}{% endfor %}{% endfor %}{% if fs.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %}  |  {% assign count = hostedcontent.size | minus: fs.size %}{% if count > 0 %}There are {{ count }} sane links{% endif %}{% if fs.size > 0 %}The {{ fs.size }} pieces with bad buckets are: {% for c in fs %}[{{ c.title | split: ":" | first }}]({{ c.url }}) {% endfor %}{% endif %}
 | file_links match formats | {% assign fs = '' | split: '' %}{% for c in hostedcontent %}{% for l in c.file_links %}{% assign ee = l | split: "." | last | downcase %}{% unless ee == c.formats[forloop.index0] or c.formats[forloop.index0] == 'mp3s' %}{% assign fs = fs | push: c %}{% endunless %}{% endfor %}{% endfor %}{% if fs.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %}  |  {% assign count = hostedcontent.size | minus: fs.size %}{% if count > 0 %}There are {{ count }} sane formats.{% endif %}{% if fs.size > 0 %} The {{ fs.size }} pieces with bad formats are: {% for c in fs %}[{{ c.title | split: ":" | first }}]({{ c.url }}) {% endfor %}{% endif %}
 | Negative Pages | {% assign fails = '' | split: '' %}{% for cont in site.content %}{% unless cont.pages %}{% continue %}{% endunless %}{% if cont.pages contains "--" %}{% assign ps = cont.pages | split: "--" %}{% assign ps = 1 | plus: ps[1] | minus: ps[0] %}{% if ps < 1 or ps != cont.page_count %}{% assign fails = fails | push: cont %}{% endif %}{% else %}{% if cont.pages < 1 %}{% assign fails = fails | push: cont %}{% endif %}{% endif %}{% endfor %}{% if fails.size == 0 %}Pass ✅{% else %}FAIL ❌{% endif %} | Found {{ fails.size }} with negative pages{% if fails.size > 0 %}: {{ fails | map: "path" | array_to_sentence_string }}{% endif %}
 | Missing Author Pages | {% assign expect = '' | split: '' %}{% for cont in site.content %}{% for auth in cont.authors %}{% unless auth contains " " %}{% unless expect contains auth %}{% assign expect = expect | push: auth %}{% endunless %}{% endunless %}{% endfor %}{% endfor %}{% assign grouped = expect | group_by_exp: "auth", "site.authors | where: 'slug', auth | size" %}{% for group in grouped %}{% if group.name == 0 %}{% assign failed = group.items %}{% else %}{% assign found = group.items %}{% endif %}{% endfor %}{% if expect.size == found.size %}Pass ✅{% else %}FAIL ❌{% endif %}  |  Of the {{ expect.size }} authors cited, {{ found.size }} of them are in the database. {% if failed.size > 0 %}Failed to find author pages for: {{ failed | array_to_sentence_string }}{% endif %}

scripts/gdrive.py

@@ -44,7 +44,10 @@
   exit(1)
 
 # If modifying these scopes, have to login again.
-SCOPES = ['https://www.googleapis.com/auth/drive','https://www.googleapis.com/auth/youtube.readonly']
+SCOPES = [
+  'https://www.googleapis.com/auth/drive',
+  'https://www.googleapis.com/auth/youtube.readonly',
+]
 # The client secrets file can be made and downloaded from your developer console:
 # https://console.developers.google.com/apis/credentials
 CLIENTSECRETS = os.path.expanduser("~/library-utils-client-secret.json")

scripts/website.py

@@ -61,6 +61,13 @@ def __init__(self, fd: Path, content, handler=None, **kwargs) -> None:
     super().__init__(fd, content, handler, **kwargs)
     self.url = "/tags/" + fd.stem
 
+class DataCollection():
+  def load(self):
+    content_config = root_folder.joinpath('_data/content.yml').read_text()
+    self.content = yaml.load(content_config, Loader=yaml.Loader)
+
+data = DataCollection()
+
 class TagCollection():
   def __init__(self):
     self.tags = dict()
@@ -99,7 +106,8 @@ def __init__(self, fd: Path, content, handler=None, **kwargs) -> None:
     fd = Path(fd)
     super().__init__(fd, content, handler, **kwargs)
     self.category = self.relative_path.parts[1]
-    self.url = f"/content/{self.category}/{fd.stem}"
+    self.content_path = f"{self.category}/{fd.stem}"
+    self.url = f"/content/{self.content_path}"
     if not self.get('tags'):
         self.tags = []
     if not self.get('formats'):
@@ -162,3 +170,4 @@ def load():
     if (not authorfile.is_file()) or authorfile.name.startswith('.'):
       continue
     authors.add(AuthorFile.load(authorfile))
+  data.load()