Language Models

This is how we select language models and model parameters for simulating survey results.

LanguageModel class

This module contains the LanguageModel class, which is an abstract base class for all language models.

class edsl.language_models.LanguageModel.LanguageModel(crud: ~edsl.data.crud.CRUDOperations = <edsl.data.crud.CRUDOperations object>, **kwargs)

Bases: RichPrintingMixin, PersistenceMixin, ABC

ABC for LLM subclasses.

property RPM

Model’s requests-per-minute limit.

property TPM

Model’s tokens-per-minute limit.

abstract async async_execute_model_call()

Execute the model call and returns the result as a coroutine.

async async_get_raw_response(user_prompt: str, system_prompt: str, iteration: int = 1) → dict[str, Any]

Handle caching of responses.

If the cache isn’t being used, it just returns a ‘fresh’ call to the LLM, but appends some tracking information to the response (using the _update_response_with_tracking method). But if cache is being used, it first checks the database to see if the response is already there. If it is, it returns the cached response, but again appends some tracking information. If it isn’t, it calls the LLM, saves the response to the database, and returns the response with tracking information.

If self.use_cache is True, then attempts to retrieve the response from the database; if not in the DB, calls the LLM and writes the response to the DB.

async async_get_response(user_prompt: str, system_prompt: str, iteration: int = 1)

Get response, parse, and return as string.

cost(raw_response: dict[str, Any]) → float

Return the dollar cost of a raw response.

classmethod example()

Return a default instance of the class.

execute_model_call(**kwargs)

classmethod from_dict(data: dict) → Type[LanguageModel]

Convert dictionary to a LanguageModel child instance.

get_raw_response(user_prompt: str, system_prompt: str, iteration: int = 1) → dict[str, Any]

Handle caching of responses.

If the cache isn’t being used, it just returns a ‘fresh’ call to the LLM, but appends some tracking information to the response (using the _update_response_with_tracking method). But if cache is being used, it first checks the database to see if the response is already there. If it is, it returns the cached response, but again appends some tracking information. If it isn’t, it calls the LLM, saves the response to the database, and returns the response with tracking information.

If self.use_cache is True, then attempts to retrieve the response from the database; if not in the DB, calls the LLM and writes the response to the DB.

get_response(user_prompt: str, system_prompt: str, iteration: int = 1)

Get response, parse, and return as string.

abstract parse_response() → str

Parse the API response and returns the response text.

What is returned by the API is model-specific and often includes meta-data that we do not need. For example, here is the results from a call to GPT-4: To actually tract the response, we need to grab data[“choices[0]”][“message”][“content”].

rich_print()

Display an object as a table.

to_dict() → dict[str, Any]

Convert instance to a dictionary.

class edsl.language_models.LanguageModel.RegisterLanguageModelsMeta(name, bases, namespace, /, **kwargs)

Bases: ABCMeta

Metaclass to register output elements in a registry i.e., those that have a parent.

REQUIRED_CLASS_ATTRIBUTES = ['_model_', '_parameters_', '_inference_service_']

static check_required_class_variables(candidate_class: LanguageModel, required_attributes: List[str] = None)

Check if a class has the required attributes.

>>> class M:
...     _model_ = "m"
...     _parameters_ = {}
>>> RegisterLanguageModelsMeta.check_required_class_variables(M, ["_model_", "_parameters_"])
>>> class M2:
...     _model_ = "m"
>>> RegisterLanguageModelsMeta.check_required_class_variables(M2, ["_model_", "_parameters_"])
Traceback (most recent call last):
...
Exception: Class M2 does not have required attribute _parameters_

classmethod get_registered_classes()

Return the registry.

classmethod model_names_to_classes()

Return a dictionary of model names to classes.

static verify_method(candidate_class: LanguageModel, method_name: str, expected_return_type: Any, required_parameters: List[tuple[str, Any]] = None, must_be_async: bool = False)

Verify that a method is defined in a class, has the correct return type, and has the correct parameters.

edsl.language_models.LanguageModel.handle_key_error(func)

Handle KeyError exceptions.