|
|
Abstract:
Letters sent as periodic updates to supporters of the Baha'i Library Online redevelopment project, which serve as a change-log.
Notes:
|
Letter 8 (2025-10-20)Baha'i Library Online redevelopment — donor letter 8
Background: Summary of Donor Letter #7, March 2025 (which was the first set of significant new features in many years).
Version 5.2 By May 2025 Version 5 was 99% completed, following its 2024 beta, its 5.1 minor reversioning, and finding and fixing a few dozen bugs from the initial 5.0 code. With the code for Tagging now mature and the infrastructure for the Inventory complete, it was time to add data. The summer was spent importing the entire Inventory database of 29,500 entries + its 661 Subjects + 1700 topics in the related Loom of Reality database + hundreds of new interlinked cross-references. We also added over a hundred Word documents created by Mike Thomas — proofread, annotated, and sometimes updated versions of older documents, both of Writings and other published books — and in many cases were able to create new and better PDF versions. By July our content was so greatly expanded that it warranted a new subversion, 5.2 (patch update 5.2.1 in October). Our content becoming interoperable is a significant upgrade: we now have common-key codes both for exporting data (specifically RIS) and for automatically linking between websites (e.g. using bahaidata's Properties). We've long envisioned a goal of a central nexus for finding Baha'i material, a comprehensive starting point for search and research. This endpoint is coming into focus!
Baha'i Library Offline, versions 5.2 and 5.2.1 With the maturation of the code and the expansion of content, it was time to make a new version of the offline site. This is the version of BLO that can be downloaded, stored, and shared — all links and content are browsable locally on your computer with no need for an internet connection. This was updated in October with hundreds of new Tags and cross-links with the Inventory. See bahai-library.com/blo_offline.
Updates to Inventory, Subjects, and Tags Inventory updates are briefly described in the section above. We also (thanks to the tireless work of A. Bolhuis) created a network of tags correlating the Inventory with the Sacred Writings, Provisional Translations, and Tag Categories. The broad Subject Categories created by Steven Phelps were also cross-referenced with our Tag cloud and then inter-linked with his "Loom of Reality" database. These 661 Subjects, now lightly integrated with our 8,500 Tags and bahaidata's 7,000 'Q' codes, forms a vast but still-growing set of "web hooks" which can be used to inter-link disparate datasets, and allow users to browse topics across multiple websites. We also went through the entire BLO catalog to add tags to a few thousand documents, and marked each and every item with either "has enough tags" or "needs more tags." The resulting list is bahai-library.com/tags/needs_tags. If you can help add Tags (a simple process of just reading documents and making note of the main topics) please contact us. Because the Inventory tables are so large — pages like bahai-library.com/inventory/ALL or bahai-library.com/inventory/subjects/spiritual_transformation had up to 30,000 links in each single page — serving these from the database would have crashed the server as soon as we had more than 2 simultaneous users. So I've been running a series of "web crawls" to convert the large database-created pages to static HTML, redoing it once a month or so as more links are added. The bandwidth usage remains large, but there's minimal impact on the server's CPU. I also crawled all the Abbreviations used in the database, such that pages like these can serve large amounts of data with minimal server drain: bahai-library.com/inventory/GWB (sources for Gleanings) or bahai-library.com/inventory/INBA (Iranian National Baha'i Archives).
Website archives Inspired by the fact that we never archived kashkul.org [archive.org] (while it’s all in Wayback Machine, that’s not crawlable so we can’t easily restore its data), I spent a couple months running web crawls of most (non-institutional or corporate) Baha'i sites with unique historical or academic content. The list is at bahai-library.com/website_archives. A few of these sites have been processed to create a locally-browsable archive such that their pages can be directly added to the bahai-library.com filesystem (should they over go offline). These local versions have been password-protected to keep out of public view while the original sites remain online.
"Jonah Winters to-do" archive As described in more detail in Bahá'í Library Online at 25, I started collecting academic and historical materials when I began academic study of the Faith around 1994. I was lucky enough to be on Talisman in its early days, when scholars were freely sharing their publications, draft papers, and various research materials. And when I started BLO in 1997, people began directly sending me their collections of work: published, unpublished, and sometimes raw data. By 1998 I already had more material than I could process and I created an online folder called "The Bin", where I simply dropped everything I received. But this only stayed online for a few months, as it became apparent that much of the material in this Bin needed to be checked for quality, accuracy, copyright, and in some cases, even anti-Baha'i or anti-Administration content. So from 1999 until 2025 this archive remained locked away on my computer, un-sharable. I chipped away at this archive over the decades, especially when I was on plane flights and had a few hours offline when I could focus on sorting and comparing documents, but it was slow going. By the 2020s there were at least 15,000 items in this "to do" folder: many duplicates, some not worth posting, some copyright. In late 2024 I made a concerted effort to "free" this material. After spending a couple months sorting for duplicates and removing scans of copyrighted books, the backlog was finally "clean" enough to share publicly. I created a "to-do / backlog" folder at archive.org/details/bahai-library-offline-files. This includes some complete data sets, like Don Calkins' huge life-long collection of biographical snippets and newspaper articles mentioning Baha'i figures, or a number of websites crawled in 2014, some of which have since gone offline but are not yet restored. It is a great relief to have this content on archive.org — both my to-do backlog and the "Baha'i Library Offline" — because it eases any fear about passing prematurely without being able to share my data. I now periodically update the files in bahai-library-offline-files, including anything sent to me that's valuable but which I'm not able to process and post yet. If I should pass tomorrow and take my computers with me, nothing will be lost. Pilgrims Notes, master list We’ve been chipping away at this gradually for years, and am nearing the completion point of creating a Master List of Pilgrim's Notes. I acquired a few sets of pilgrims notes collections over the past 30 years, as it’s the kind of thing Baha’is like to collect. They needed to be sorted for duplicates, corrected versions, formatted versions, original scans, etc. The resulting table is at bahai-library.com/pilgrim_accounts_collection. This set was compared against what’s online now at bahai-library.com/Pilgrims and bahai-library.com/tags/pilgrims_notes. A team of three is now working on bahai-library.com/pilgrims-notes_us_archives_1898-1959, which will be compared against the first two sets linked above. From an initial set of about 1,000 unsorted documents, I only have 12 pilgrims notes left to post. Some of these could be unique: not only never previously online, but also not known to the other collectors.
Internet attacks: DDoS and volumetric attacks Unfortunately, in September we dealt with a bandwidth-hogging "volumetric" attack, then in October we had our first targeted DDoS attack. The bandwidth volume issue was solved first by blocking the single offending IP — it could have been a web-crawling script gone rogue, or an individual without hacking skills attempting to take the Library offline. Had this not been caught in time, all the sites on the server would have gone offline until the end of the month. I went through the logs to find a number of places to tune the HTML so that we wouldn't get hit so hard if it happens again. One culprit was the new page bahai-library.com/inventory/ALL* and many of the other text- and link-heavy Inventory pages, which I cut in half. I also offloaded all large files, 100MB+ (mostly zip, some pdf and mp3 files), to archive.org. The only downside is that some were PDFs which now won’t be Google-indexed under our domain name, but most of these had optimized duplicate versions anyway, which remain local and hence indexed. Resolving the DDoS (Distributed Denial of Service) issue was trickier. It was a direct attack on bahai-library.com and only on bahai-library.com, using a range of IPs from Singapore. As we got no ransomware emails, I presume it was from an individual or a nation-state opposed to the Faith. My datacenter’s tech support was able to help by reconfiguring our firewall. For a week I closely watched the live server traffic to see which IPs to block, updated the firewall, and manually restarted the web-server repeatedly, sometimes every 10 minutes for hours. After a few days of that we’d gotten all their IPs, and then it was just another week of watching the server logs.
New listserver domain bahai-library-lists.org Another datacenter surprise came in August: an email block on our mailing lists. To introduce this topic, let me explain that we have two servers. My personal and commercial accounts are on a "virtual shared server" but BLO is on its own "virtual private server." The shared server has the advantages of being cheap, and managed by tech support; its disadvantages are that it can be slow, and I have no control over its settings. This is the server which housed bahai-library-lists.com and all of our email lists, both the private ones for academics and historians and the public ones for general scholarly discussion. The datacenter upgraded to a new operating system in August and, with no prior warning to account holders, imposed various anti-spam measures, one of which limits domains to 150 emails per hour. This meant that Tarikh (the current incarnation of the original Talisman), our most active list, began silently discarding emails; then, a later update automatically disabled all email sending for all lists once that limit was hit. The simplest solution was to register a new domain and create a new account, which I did with bahai-library-lists.org. While I could simply point the old domain to the new server, we would then lose the 2023-2025 archives. I can’t easily migrate Mailman, sadly; it is not like an easily-portable MySQL database, for technical reasons I won't explain here. Of interest: part of why we keep registering new domains and creating new Mailman installations is so that no prior archives are lost when we move to new servers or new software. We still have the archives going back to 2003, back when the lists were using Lyris on the University of Buffalo network!
Near-term updates In the remaining weeks of 2025, we will:
Version 6 and Long-term plans, 2026 With the codebase now so large and interconnected (the 5.2 code is roughly three times the size of 4.6 and the database has three times as many tables), I'm now considering a simpler refactoring of 5 rather than a wholesale rewrite. Given the extensive functions of the current code, it would be a prohibitively expensive job if we were to hire a developer. By the time new bugs were fixed and all new functions working, it took about 19 months of largely full-time work to create 5 (including data and import for Tags, Inventory, and Q Properties). But its growth over that time was organic, with new functions tacked-on rather than being properly incorporated from day one. Version 5 was coded in spring 2024 as an emergency stop-gap solution when our 2022-2023 redevelopment project failed and the site was just one month away from being taken offline permanently (the server and its OS were both past end-of-life). Its initial implementation prioritized speed over quality: its framework is not logical, its code full of duplication, and its admin interface inconsistent. I propose to take the 500 pieces of code and functions that now comprise 5, lay them out on the table so to speak, create a flowchart of the stack (user interface → PHP → database → PHP → user), and re-organize everything coherently. At that time we can also redo the CSS for proper display on mobile. There were only two real reasons to consider a professionally-coded site — (1) maintenance without me, and (2) security — because: 1) My amateur code works, but wouldn't be scalable by someone else if I pass unexpectedly, due to its spaghetti nature. 2) I don't know how to write secure code. But these are not important matters as long as I'm around. For security, I simply monitor Apache status and server processes to see which pages are vulnerable, watch for cross-site scripting, block IPs, make fixes as needed, and make regular backups. A refactor of 5 (rather than a complete rewrite as 6) will partially resolve both issues without having to invest a likely 2 years and $20,000+ on a professional redevelopment; my messy codebase can be organized coherently (with extensive commenting), and I can better limit vulnerabilities. And, after doing this the last 19 months, I’m more familiar with PHP 8.
Letter 7 (2025-03-22)Baha'i Library Online redevelopment — donor letter 7 Dear friends,
Summary: Version 5.1 is complete! Citation export is done, advanced search has been upgraded, the Partial Inventory infrastructure has been incorporated, bahaidata.org has been incorporated, and a user/admin interface has been created.We can now begin inviting the public, including you receiving this email, to add data to the Library — starting with adding Tags and Chronology events. Email me if you’d like to help.
Background: Version 5.0 beta, August 2024, was our initial, minimal-but-functioning migration from the previous expired server and its end-of-life code.Version 5.09, December 2024, was essentially same as Version 4.5 but now fully refactored, with beta bugs fixed and all functions replaced (and a new Author and Tag system). Version 5.1, March 2025, is the first set of significant new features.
Details: 1) Citation export, bahai-library.com/ris — Our content can now be integrated with other databases and reference apps via RIS.This has been a goal since 2002: standardizing our content using some form of a “common standard”, and creating an interface for database interoperability. Now for the first time, bibliographers can integrate BLO content. RIS records can be exported from any file, from any Collection (see bottom of any page), and from user-specified Advanced Search queries (example below). 2) Advanced search, bahai-library.com/advancedsearch — granular search has been a long-needed feature. Individual searches also provide a sharable web link. Here are some sample uses:
One advantage of this system is it allows for a single, stable web link for every Tablet, e.g. the Persian Bayan bahai-library.com/inventory/BB00001 , which can be cross-referenced with each Tablet’s Tag page, bahai-library.com/tags/Bayan-i-Farsi_(Persian_Bayan) 4) We have also begun incorporating the new site bahaidata.org, which is a central repository of links and data for use by the Baha’i wiki sites; see bahai-library.com/Properties. For example, at the bottom of the above link bahai-library.com/tags/Bayan-i-Farsi_(Persian_Bayan) see the number “Q1002” and "Links to Bahá'í wikis.” We can, using this "webhook,” integrate our tag cloud with the wiki network. 5) User/admin interface — with a GUI in place now, we can invite people to join the site to add tags, add events to the Chronology, and upload files, either their own files or to help us get through our backlog of 14,000 items waiting to be checked or uploaded. We especially need help uploading documents in Persian. Please email jonahwinters@gmail.com if you’d like to join our team, and forward this invite to anyone you know who might be interested. These are the last major upgrades planned, pending the hand-over to a future Version 6 team (who can hopefully recreate what I’ve coded, but with professional syntax).
-Jonah Winters, on behalf of Twenty-seven Letters Foundation Letter 6 (2025-01-14)Baha'i Library Online redevelopment — donor letter 6 Dear friends, 1) The remainder of autumn 2024, since our last email, was spent updating the Library with new content, expanding the Tags cloud, and fixing a number of bugs and patching holes, by the end of December completing Version 5.09 (bug fixes). More details are at bahai-library.com/version_5 2) As a culminating task, we made a new (and improved) version of Baha’i Library Offline, which can now be downloaded via bahai-library.com/blo_offline (BitTorrent can be fastest). Please download that, plus bahai-library.com/blo_database , to ensure we have multiple backup copies around the world. 3) The first new feature, hence Version 5.1, is the long-requested Advanced Search function: bahai-library.com/advancedsearch The next features in progress are citation export, and creating user-specific functions to share editing and data input. These are the last two upgrades planned, pending the hand-over to a Version 6 team, to be created later in 2025. Letter 5 (2024-09-30)Baha'i Library Online redevelopment — donor letter 5 Dear friends, Version 5 of Baha'i Library online is done! This email summarizes (1) completed updates, (2) upcoming changes, and (3) two things we need help with.
1) Completed updates; site status Like the previous four versions, this was a ground-up rewrite.All functions of the previous website have now been restored, and it's working on mobile phones. The live site is back at the primary domain bahai-library.com, and the April 2024 archive has been removed. This follows our longest stable release period — Version 4 had been in place for over 12 years! (Version one was 1997-2002; two 2003-2004; three 2005-2011; four 2012-2024.) Features restored include: Chronologies and search interfaces; Date, Title, and Author search; Series; Languages; RSS; Site map; font size script; Google search; all admin functions; links updated. New features include:
2) To do
3) Need help with
Letter 4 (2024-07-19)Baha'i Library Online redevelopment — donor letter 4 Dear friends, It’s only been six weeks since the last letter, but things have come into better focus in that time. This email describes (1) the current site and (2) future plans. These same details are at bahai-library.com/version_5 and bahai-library.com/version_6
1) Current site:
2) Future site:
Letter 3 (2024-06-03)Baha'i Library Online redevelopment — donor letter 3 Dear friends, With appreciation, thank you for your support for the Baha’i Library Online and its ongoing redevelopment project. Plans were delayed for a bit, but we've remained in constant progress behind-the-scenes, and have just started a new phase. Some details are at bahai-library.com/version_5. Here’s a summary. As mentioned in the last email, we’re now a registered 501(c)(3) non-profit group “Twenty-seven Letters Foundation, Inc.” We have a dedicated bank account, and as of February we now also have tax-exempt status. Our transition to the cloud on AWS ended up not working at this time. Instead, we have just moved to a new Virtual Private Server and are redeveloping the site using the same software stack it’s been running since 2003: PHP and MySQL on a private server. This is for the best, as PHP continues to be a viable solution for our website's needs, and easier to build than a cloud implementation. For now and until the new PHP coding is completed, the website bahai-library.com is a static archive, in plain HTML, showing the last incarnation of Version 4.5. It has also been freshly archived in Wayback Machine. We just completed the move to a new server and created an interim interface using the domain bahai-library.org . This will remain the “live” site, with updates and recent content, until we create a new version. We have begun interviewing coders; if you know someone with experience in PHP, please send me their name. Letter 2 (2024-01-20)Baha'i Library Online redevelopment project — second donor letter Dear friends, You are receiving this email because you have donated to the Baha’i Library Online redevelopment fund, or otherwise expressed interest in the project. This is a brief update. At the end are some links from our first letter. We will send an email with tech-related updates next month.
c/o Jonah Winters 12 Tasker Street St. Catharines, ON, L2R 3Z8 Canada (note: postage from the US to Canada is currently $1.60)
- Alexander, Jonah, and Michael Letter 1 (2023-09-21)Baha'i Library Online redevelopment project — first donor letter You are receiving this email because you donated to the Baha’i Library Online redevelopment fund earlier this year, or otherwise expressed interest in the project. We do not yet have any groundbreaking news, but progress has been steady (if slow). When the registration for our 501(c)(3) non-profit registration has been completed, we will email you again with more information. This is a short email to let you know that work on the project has been steady throughout 2023, and your donation was very needed and appreciated.
-Jonah
|
| METADATA | |
| Views | 562 views since posted 2025-10-22; last edit 2025-10-26 00:45 UTC; previous at archive.org.../donor_letters |
| Language | English |
| Permission | author |
| Share | Shortlink: bahai-library.com/7097 Citation: ris/7097 |
|
|
|
Home
search Author Adv. search Links |
|
|