| Title: | Cleaning Text Data with an AI Assistant |
|---|---|
| Description: | Provides functions to clean and standardize messy data, including textual categories and free-text addresses, using Large Language Models. The package corrects typos, expands abbreviations, and maps inconsistent entries to standardized values. Ideal for Bioinformatics, business, and general data cleaning tasks. |
| Authors: | Emi Tanaka [aut, cph] (ORCID: <https://orcid.org/0000-0002-1455-259X>), Jiajia Li [aut, cre] (ORCID: <https://orcid.org/0009-0003-7143-9336>) |
| Maintainer: | Jiajia Li <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-07-05 08:33:49 UTC |
| Source: | https://github.com/anuopensci/emend |
A sample dataset of Airbnb listings and reviews of properties from Sydney, Australia.
airbnb_listings airbnb_reviewsairbnb_listings airbnb_reviews
airbnb_listingsA data.frame with 1623 rows and 68 columns
Airbnb's unique identifier for the listing.
Name of the listing.
Detailed description of the listing.
Host's description of the neighbourhood.
URL to the Airbnb hosted regular sized image for the listing.
Airbnb's unique identifier for the host/user.
Name of the host. Usually just the first name(s).
The date the host/user was created. For hosts that are Airbnb guests this could be the date they registered as a guest.
The host's self reported location.
Description about the host.
The time interval between when a host responds to an inquiry from a guest.
Percentage of inquiries from potential guests that are responded to by hosts.
That rate at which a host accepts booking requests.
Whether the host is a super host or not.
A thumbnail of the host.
A URL to the picture of the host.
The host neighbourhood.
The number of listings the host has.
The number of listings the host has.
Host communication verifications.
Whether the host has a profile pic.
Whether the host has their identity verified.
The neighbourhood as geocoded using the latitude and longitude against neighborhoods as defined by open or public digital shapefiles.
Uses the World Geodetic System (WGS84) projection for latitude and longitude.
Uses the World Geodetic System (WGS84) projection for latitude and longitude.
Self selected property type. Hotels and Bed and Breakfasts are described as such by their hosts in this field.
Entire home/apt, Private room, Shared room, or Hotel. Entire places are best if you're seeking a home away from home. With an entire place, you'll have the whole space to yourself. This usually includes a bedroom, a bathroom, a kitchen, and a separate, dedicated entrance. Hosts should note in the description if they'll be on the property or not (ex: "Host occupies first floor of the home"), and provide further details on the listing. Private rooms are great for when you prefer a little privacy, and still value a local connection. When you book a private room, you'll have your own private room for sleeping and may share some spaces with others. You might need to walk through indoor spaces that another host or guest may occupy to get to your room. Shared rooms are for when you don't mind sharing a space with others. When you book a shared room, you'll be sleeping in a space that is shared with others and share the entire space with other people. Shared rooms are popular among flexible travelers looking for new friends and budget-friendly stays.
The maximum capacity of the listing.
The number of bathrooms in the listing.
The text of the number of bathsroom in the listings.
The number of bedrooms.
The number of bed(s).
The amenities.
Daily price in local currency.
Minimum number of night stay for the listing.
Maximum number of night stay for the listing.
The smallest minimum_night value from the calender (looking 365 nights in the future).
The largest minimum_night value from the calender (looking 365 nights in the future).
The smallest maximum_night value from the calender (looking 365 nights in the future).
The largest maximum_night value from the calender (looking 365 nights in the future).
The average minimum_night value from the calender (looking 365 nights in the future).
The average maximum_night value from the calender (looking 365 nights in the future).
Whether there is availability or not.
The availability of the listing x days in the future as determined by the calendar. Note a listing may not be available because it has been booked by a guest or blocked by the host.
The number of reviews the listing has.
The number of reviews the listing has (in the last 12 months).
The number of reviews the listing has (in the last 30 days).
The date of the first/oldest review.
The date of the last/newest review.
The review score for ratings of the listing.
The review score for accuracy of the listing.
The review score for cleanliness of the listing.
The review score for checkin experience of the listing.
The review score for communication of the listing.
The review score for location of the listing.
The review score for value of the listing.
The licence/permit/registration number.
Whether the guest can automatically book the listing without the host requiring to accept their booking request. An indicator of a commercial listing.
The number of listings the host has in the current scrape, in the city/region geography.
The number of Entire home/apt listings the host has in the current scrape, in the city/region geography.
The number of Private room listings the host has in the current scrape, in the city/region geography.
The number of Shared room listings the host has in the current scrape, in the city/region geography.
The average number of reviews per month the listing has over the lifetime of the listing.
airbnb_reviewsA data.frame with 5679 rows and 6 columns
Unique identifier for the listing
Unique identifier for the review
Date of the review
Unique identifier for the reviewer
Name of the reviewer
Text of the review
https://insideairbnb.com/get-the-data/
Year
Month
Supplier
Item code
Item description
Item type
Retail sales
Retail transfers
Warehouse sales
alcoholalcohol
An object of class spec_tbl_df (inherits from tbl_df, tbl, data.frame) with 1000 rows and 9 columns.
https://catalog.data.gov/dataset/warehouse-and-retail-sales
A sample of reviews from Amazon in the applicances category.
consumerconsumer
A data.frame with 4549 rows and 11 columns
Overall rating of the product.
Whether the reviewer is verified or not.
Review date.
A unique identifier of the reviewer.
The name of the reviewer.
The text of the review.
Summary of the review.
The number of helpful votes of the review.
The images that the reviewer post after they receive the product.
Jianmo Ni, Jiacheng Li, Julian McAuley (2019) Justifying recommendations using distantly-labeled reviews and fined-grained aspects. Empirical Methods in Natural Language Processing .
This function standardise inconsistent address formats to a standard format.
emend_clean_address(address_vector, chat = get_default_chat())emend_clean_address(address_vector, chat = get_default_chat())
address_vector |
A character vector that is assumed to be addresses. |
chat |
A chat object defined by ellmer. |
A character vector with converted addresses.
# Convert a vector of inconsistent formatted address to a standard format options(ellmer_timeout_s = 3600) x <- c("154 university avenue, acton act 2601", "76/2 Cape Street, Dickson ACT 2602", "Shop 4/96 Bunda St, Canberra ACT 2601", "11 E Row, Canberra ACT 2601", "173/46 Macquarie St, Barton ACT 2600", "Unit 189/260 City walk, Canberra ACT 2601", "the kebab place", "i don't know the address") chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_clean_address(x, chat = chat)# Convert a vector of inconsistent formatted address to a standard format options(ellmer_timeout_s = 3600) x <- c("154 university avenue, acton act 2601", "76/2 Cape Street, Dickson ACT 2602", "Shop 4/96 Bunda St, Canberra ACT 2601", "11 E Row, Canberra ACT 2601", "173/46 Macquarie St, Barton ACT 2600", "Unit 189/260 City walk, Canberra ACT 2601", "the kebab place", "i don't know the address") chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_clean_address(x, chat = chat)
This function standardise inconsistent date formats.
emend_clean_date(dates_vector, chat = get_default_chat())emend_clean_date(dates_vector, chat = get_default_chat())
dates_vector |
A character vector that is assumed to be dates. |
chat |
A chat object defined by ellmer. |
A vector of Date objects.
x <- c("16/02/1997", "20 November 2024", "24 Mar 2022", "2000-01-01", "Jason", "Dec 25, 2030", "11/05/2024", "March 10, 1999") chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_clean_date(x, chat = chat)x <- c("16/02/1997", "20 November 2024", "24 Mar 2022", "2000-01-01", "Jason", "Dec 25, 2030", "11/05/2024", "March 10, 1999") chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_clean_date(x, chat = chat)
Match input factor to specified levels.
emend_fct_match(.f, levels = NULL, chat = get_default_chat())emend_fct_match(.f, levels = NULL, chat = get_default_chat())
.f |
A factor. |
levels |
The levels of the factor |
chat |
A chat object defined by ellmer. |
A factor with levels matching the provided levels argument.
chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_fct_match(messy$country, levels = c("UK", "USA", "Canada", "Australia", "NZ"), chat = chat)chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_fct_match(messy$country, levels = c("UK", "USA", "Canada", "Australia", "NZ"), chat = chat)
Reorder the levels of the input factor in a meaningful way.
emend_fct_reorder(.f, chat = get_default_chat())emend_fct_reorder(.f, chat = get_default_chat())
.f |
A vector of characters or a factor. |
chat |
A chat object defined by ellmer. |
A factor with standardized category labels.
chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_fct_reorder(likerts$likert1, chat = chat) |> levels()chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_fct_reorder(likerts$likert1, chat = chat) |> levels()
Match the input factor to supplied levels.
emend_lvl_match(.f, levels = NULL, chat = get_default_chat())emend_lvl_match(.f, levels = NULL, chat = get_default_chat())
.f |
A vector of characters or a factor. |
levels |
The levels of the factor. |
chat |
The chat object defined by ellmer. |
A named character vector of standardised category labels, with the class "emend_lvl_match".
The names correspond to the original messy categories, and the values are the cleaned versions.
chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_lvl_match(messy$country, levels = c("Asia", "Europe", "North America", "Oceania", "South America"), chat = chat)chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_lvl_match(messy$country, levels = c("Asia", "Europe", "North America", "Oceania", "South America"), chat = chat)
The returned value is a vector. The LLM will return full names instead of abbreviations. You can use this functions to clean up your categorical data and obtain unique levels. Double check if the output from LLM is true to your data. This function is generally suitable for categories, not working well with sentences and too many categories.
emend_lvl_unique(.f, chat = get_default_chat())emend_lvl_unique(.f, chat = get_default_chat())
.f |
A vector of characters or a factor. |
chat |
A chat object defined by ellmer. |
A character vector of standardised category names.
options(ellmer_timeout_s = 3600) chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_lvl_unique(messy$country, chat = chat)options(ellmer_timeout_s = 3600) chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_lvl_unique(messy$country, chat = chat)
Translate text from one language to another.
emend_translate(text, to = "English", chat = get_default_chat())emend_translate(text, to = "English", chat = get_default_chat())
text |
The text to translate. |
to |
The language to translate to. The default is "English". |
chat |
An ellmer Chat object. |
A character vector of translated text.
chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_translate(c("\u733F\u3082\u6728\u304B\u3089\u843D\u3061\u308B", "\u4F60\u597D", "bon appetit"), chat = chat)chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_translate(c("\u733F\u3082\u6728\u304B\u3089\u843D\u3061\u308B", "\u4F60\u597D", "bon appetit"), chat = chat)
Identify the language in the text.
emend_what_language(text, chat = get_default_chat())emend_what_language(text, chat = get_default_chat())
text |
A string or a factor that contains text information. |
chat |
A chat object defined by ellmer |
A character vector of language names.
chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_what_language(c("\u733F\u3082\u6728\u304B\u3089\u843D\u3061\u308B", "\u4F60\u597D", "bon appetit"), chat = chat)chat <- ellmer::chat_ollama(model = "llama3.1:8b", seed = 0, echo = "none") emend_what_language(c("\u733F\u3082\u6728\u304B\u3089\u843D\u3061\u308B", "\u4F60\u597D", "bon appetit"), chat = chat)
Get or create the default chat object
get_default_chat()get_default_chat()
A Chat object.
A sample review of hotels.
hotelhotel
A data.frame with 13,193 rows and 8 columns
The name of the reviewer.
The nationality of the reviewer.
Reviewer's rating.
Date of the review.
Title of the review.
Text of the review.
Name of the hotel being reviewed.
Average rating of the hotel.
https://www.kaggle.com/datasets/nikitaryabukhin/reviewshotel
A data set containing 9 different likert scales.
likertslikerts
A data.frame with 40 rows and 9 columns
A 7-point agreeableness likert scale.
A 5-point agreeableness likert scale.
A 5-point agreeableness likert scale as a sentence.
A 5-point frequency likert scale.
A 5-point rating likert scale.
A 5-point likelihood likert scale.
A 5-point likert scale.
A 5-point satisfaction likert scale.
A 6-point priority likert scale.
A synthetic dataset that contains inputs with some common standardisation issues.
messymessy
A list of 3 character vectors
A character vector of countries.
A character vector of suburbs in Australia with various typos.
A character vector of schools or college (with typos) at the Australian National University.
A sample of 200 recipes.
recipesrecipes
A data.frame with 200 rows and 7 columns.
The name of the dish.
The list of ingredients.
The URL of the recipe.
An image of the dish.
Cooking time.
Preperation time.
The number of servings in text.
https://github.com/jakevdp/open-recipe-data/tree/main
A dataset containing registration information for a series of workshops The dataset includes columns for workshop name, affiliation, and the cleaned affiliation by hand.
registrationregistration
A data frame with 221 rows and 3 columns:
Character: Workshop name and time.
Character: Affiliation input by participants.
Character: The cleaned affiliation type, cleaned manually by author.
Real registration data from the Biological Data Science Institute, Australian National University.
Iori is a Japanese restaurant in Canberra, Australia. The data contains a sample of 20 reviews from Google maps.
restaurantrestaurant
A data.frame with 20 rows and 2 columns
The text of the review.
A rating out of 5.
Ask A Manager Salary Survey 2021
salarysalary
A data.frame with 28,083 rows and 18 columns
The timestampe of response.
Age cateogry. "How old are you?"
Categorical but respondents can enter text for Other. "What industry do you work in?"
Text entry. "Job title"
Text entry. "If your job title needs additional context, please clarify here:"
Text entry. "What is your annual salary? (You'll indicate the currency in a later question. If you are part-time or hourly, please enter an annualized equivalent – what you would earn if you worked the job 40 hours a week, 52 weeks a year.)"
Text entry. "How much additional monetary compensation do you get, if any (for example, bonuses or overtime in an average year)? Please only include monetary compensation here, not the value of benefits."
Categorical entry. "Please indicate the currency"
Text entry. "If "Other," please indicate the currency here:"
Text entry. "If your income needs additional context, please provide it here:"
Text entry. "What country do you work in?"
Categorical entry. "If you're in the U.S., what state do you work in?"
Text entry. "What city do you work in?"
Categorical entry. "How many years of professional work experience do you have overall?"
Categorical entry. "How many years of professional work experience do you have in your field?"
Categorical entry."What is your highest level of education completed?"
Categorical entry. "What is your gender?"
Multiselect entry. "What is your race? (Choose all that apply.)"
https://www.askamanager.org/2021/04/how-much-money-do-you-make-4.html