I have used and benefited from the "getbib" script and the instructions on LaTeX from Luke for a long time. So, I have put a lot of thought into this script, since I am very interested in academia. Hope you all like this.
Justifications for Improvements
This script stands out as a highly valuable (at least in my opinion) and efficient tool for managing and fetching BibTeX entries for DOIs found in PDF files or provided directly. The robust design and comprehensive functionality make it an indispensable asset for researchers. The main reasons for its superiority are as follows:
- Exceptional time-saving: By automating the process of extracting DOIs and fetching BibTeX entries, the script drastically reduces the manual effort involved in managing citations, thereby saving users an incredible amount of time and energy.
- Outstanding versatility: The script's ability to handle various input types, including directories containing PDF files, single PDF files, and DOIs, sets it apart from other solutions. This adaptability allows users to process numerous scenarios with ease, making it the go-to tool for all their citation needs.
- Unparalleled consistency: The script ensures that DOIs are uniformly processed and normalized, improving the consistency of the entries in the BibTeX file. This feature is crucial for maintaining a clean and professional bibliography that adheres to high academic standards. It inserts an empty line between entries inside the BIB_FILE, as well as, making the author name lower case. It also removes any special characters and the first 2 numbers of the year from the first line. So it is easier to read, maintain and easier to use inside a LaTeX document. Normalizing also helps to check for duplicate entries. It prevents some weird entries escaping from getting caught as a duplicate.
- Remarkable duplicate prevention: The script's built-in functionality to check for duplicate entries before appending them to the BibTeX file demonstrates a keen attention to detail. This feature ensures that the bibliography remains free of redundancies, streamlining the citation management process.
- The use of functions and modular design in the script makes the code highly readable, maintainable, and extendable. This strong foundation allows for seamless adaptation to future changes and requirements.
- Provides users with an exceptional level of automation, versatility, and reliability.
- You can provide the DOI address even in very wrong forms and get a correct output. You can even feed it a website URL such as: https://doi.org/10.1038/s41594-023-00968-y and all of the DOI handling is done by a single "sed" command.
- Robust notification system to learn more about the errors or other types of feedback.
- The "curl" output is in red in order to separate the output and the notification better and to improve readability.
Details
BIB_FILE: The path to the BibTeX file where entries will be saved.
CORRECTION_METHOD: A very powerful sed command to extract and correct the DOI from the input even in harsher cases.
get_doi_from_pdf function: Extracts a DOI from the provided PDF file using pdfinfo and pdftotext commands.
If pdfinfo doesn't find a DOI, it uses pdftotext to extract it from the first page of the PDF.
normalize_doi function: Normalizes the DOI by converting it to lowercase.
process_doi function: Fetches the BibTeX entry for the given DOI using the Crossref with a curl command.
Prints the output of the curl command in red using ANSI escape codes.
Checks if the fetched BibTeX entry is valid and not empty.
If the fetched BibTeX entry is not in the BIB_FILE, it appends the entry to the file.
The script processes input arguments, which can be a directory, a PDF file, or a DOI:
a) If it's a directory, the script processes all PDF files in the directory.
b) If it's a PDF file, the script processes the single PDF file.
c) If it's a DOI, the script processes the DOI directly.
More details on the correction method (sed command), from my prior pull request
Very Detailed Explanation (I realized that escaped backslashes do not appear. There is a backslash if you see nothing.)
(For people who wonder about it, or try to learn. It could take a tremendous amount of time to learn all of it without explanation, so it would be better to explain):
sed The sed command is a stream editor that can be used to perform basic text transformations on an input file or from a pipeline. You can see Luke uses it a lot in his videos. It can also modify files' content if you want for other purposes. That function is used a lot for bootstrapping scripts for changing config files automatically if necessary.
-n This option tells sed not to print lines by default. We'll only print lines when we specify the p command in the script.
-E This option enables the use of extended regular expressions, which allows for more readable and flexible regex patterns.
's/ This starts the sed script and defines the s command (substitute). It is used to find a regex pattern in the input and replace it with a specified string.
.* This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all characters before "doi" or "DOI".
( This paranthesis opens a capturing group, which allows us to refer back to the matched text later in the script.
(DOI|doi) This regex pattern matches either "DOI" or "doi". The | symbol is used as an OR operator in regular expressions.
( This next paranthesis opens another capturing group.
(.(org))? This regex pattern matches an optional ".org". The . is an escaped period, and (org) matches the string "org". The ? following the group makes it optional. Escaping is needed for most of non-alphanumeric characters. You can test and practice them on vim, trying to use the "substitute" function to change some text.
/? This regex pattern matches an optional "/", with the ? making it optional. The prior backslash is for escaping. Again, some characters need to be escaped to be able to used in commands. Escaped means they have ** before them. Spaces may be the most escaped characters.
| This symbol, later, also acts as an OR operator, indicating that the pattern before or after it can be matched.
**:? *** This regex pattern matches an optional colon (":") followed by zero or more spaces. The ? makes the colon optional, and ***** matches zero or more spaces.
) This closes the capturing group started earlier.
) This closes the outer capturing group.
([^: ]+[^ .]) This regex pattern matches any character except colons and spaces one or more times ([^: ]+) Plus symbol here shows one or more times. If it is a star then it means zero or more times. It is then followed by a single alphanumeric character ([^ .]) Single because there are no plus or star symbol next to it. This part as a whole ensures that the last character of the matched text is alphanumeric.
.* This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all remaining characters in the input line.
/ This delimiter separates the regex pattern from the replacement string in the s command. s command needs a separator that is a forward slash.
doi:\6 This is the replacement string. The text "doi:" is followed by the 6th captured group from the regex pattern, which contains the characters after "doi" or "DOI" and the colon, "/", or space(s).
/p This delimiter separates the replacement string from the p command, which tells sed to print the modified line if a substitution has been made. The substitution mentioned here is the change of ".org/" to ":". This helps turning URLs into doi addresses.
; This separates different commands within the sed script.
T This command branches to the end of the script if no substitution was made since the last input line was read or conditional branch was taken. In this case, it ensures that the q command is only executed if a matching line has been found and a substitution was made. This is one of the most important parts to get the doi address from the urls such as "https://doi.org/10.1038/s41594-023-00968-5". Because we don't always have URLs for doi addresses. In this way, this function only works when we work with URLs. So in this case it helps changing .org/ with : This makes the part of the doi address as this: "doi:" rather than this: "doi.org/".
q This command tells sed to quit processing after the first match, ensuring that only the first matching line in the file is processed. Otherwise, we would get all doi addresses in a scientific study because there are lots of doi addresses in them.
' This closes the other '
TL;DR:
Basically this whole command ensures that the output we get starts with "doi:", then it can have every type of character in it except spaces and ".org/" , then it will end with an alphanumeric character [A-Z, a-z or 0-9]. That ensures removing the trailing dots from some doi addresses that have them.
In the current update, the libreoffice package doesn't have any binary for libreoffice writer, calc, impress, draw, etc. So it's better to open all of the document using the 'libreoffice' binary as it can work in the old version of libreoffice as well as the new one.
