diff --git a/Makefile b/Makefile index 691efc3..7361b25 100644 --- a/Makefile +++ b/Makefile @@ -17,4 +17,4 @@ clean: find . -name '.pytest_cache' -exec rm -rf {} + find . -name '__pycache__' -exec rm -rf {} + find . -name 'debug.txt' -exec rm -rf {} + - find . -name 'debug.txt' -exec rm -rf {} + + diff --git a/docs/source/blog/blog.rst b/docs/source/blog/blog.rst index 50ebcc6..2236e71 100644 --- a/docs/source/blog/blog.rst +++ b/docs/source/blog/blog.rst @@ -1093,3 +1093,37 @@ Blog """""" Hi, I have the feeling that I am almost there with the parsers part. I have + + + +2021-01-30 +********** + +Blog +"""""" + +It's been a very long time since I wrote in the blog. I guess I was too busy to spend some time on it. +2020 was the year of the COVID 19. I guess that being contained with my son and my wife gave me other priorities. + +Nevertheless, I kept on working on Sheerka. + +One major achivement that I made was a demonstration of Sheerka to other people. I happened in mid September. +I demonstrated how to declare numbers using the ExactParser and the BnfNodeParser. Even thought I did not explicitly +mentionned theses names, my demo was far to technique. The impression I gave was that Sheerka was too complicated (in +the sens of too technical too use). Unfortunately, that was the case. + +Did the demo come too soon ? I guess so, in a certain way, but It really helped me to have a first feedback, even if it did +not reward the months of hard works. + +Where am I today ? It's impressive the admit that since that demon (in September) I did not implemented any major capability. +Sheerka does not know much bettet that at this time + +I did though work on : + + * performances + * debugger + * simple version of the rule engine (that goes with the debuggger) + +Actually almost FOUR months of work for technical benefit. There were some parts of the code that were rewritten. + +I am a little bit sad, time flies so fast. \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index a229c9d..4183922 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -26,6 +26,7 @@ There will be two types of documentation tech/tech tech/debugger + tech/rules diff --git a/docs/source/tech/debugger.rst b/docs/source/tech/debugger.rst index faa0f97..b7e0cd5 100644 --- a/docs/source/tech/debugger.rst +++ b/docs/source/tech/debugger.rst @@ -17,3 +17,119 @@ in an 'imperative' way. First, you need to enable the debugger. As it consumes resource, it is deactivated by default :: set_debug() + + +There are three types of objects that can be debugged : + + * variables + * concepts + * rules + +Debug Settings +-------------- + +Path to object +************** +Whatever the type of object to be displayed, it lies within a method name, which is itself inside a service. + +For example, the out_tree (the ast tree that contains what to print once an input is evaluated) is under the service +'Out' and within the method 'create_out_tree' + +The information regarding the evaluation of the rule #1 is under the service 'EvaluateRules', and within the method 'evaluate_rule' + +Context id +********** +The same object may be requested several times. To distinguish different execution, the ExecutionContext id is used. +When using the context id, you can precise if you want to debug a specific context, or a context and its children + +Debug id +******** +The debug id is supposed to regroup information of the same unit of processing together. For example, within the same context, +if a specific piece of processing is called several times (because of a loop for example), they will share the same context id, +but they will have different debug id. + +Note that as of now, this does not +work very well. This information need to be tuned. + +Debug Variables +--------------- + +Debugging variables let you see the content of variables, but also the execution flow of a piece of program. +This execution flow gives a context to the variables + +To activate variable debugging : + + debug_var(, , ) + + +With: + + * path_to_variable + * 'service_name' : Activates the debug logs for all methods of the service. The variable won't be shown + * 'service_name.method_name' : Activates the debug logs for the specific method of the service, the variables won't be shown + * 'service_name.method_name.variable_name' : Activates the display of a variable, within a specific method, for a specific service + * 'service_name.*' + + * context_id + * context_id + * 'context_id+' (context_id followed by the sign '+') + + * debug_id + * debug_id + +Debug Rule +--------------- + +Debugging rules let you see how the rules are evaluated, but also the execution flow of the evaluation. + +To activate rule debugging : + + debug_var(, , ) + + +With: + + * path_to_rule (when a string is given) + * 'service_name' : Activates the debug logs for all methods of the service. The variable won't be shown + * 'service_name.method_name' : Activates the debug logs for the specific method of the service, the variables won't be shown + * 'service_name.method_name.rule_id' : Activates the debug of a specific rule, within a specific method, for a specific service + * 'service_name.*' + + * path_to_rule (when an integer is given) + * rule_id + + * context_id + * context_id + * 'context_id+' (context_id followed by the sign '+') + + * debug_id + * debug_id + + +Debug Concept +--------------- + +Debugging concept let you see how the concepts are evaluated, but also the execution flow of the evaluation. + +To activate concept debugging : + + debug_var(, , ) + + +With: + + * path_to_concept when a string is given + * 'service_name' : Activates the debug logs for all methods of the service. The variable won't be shown + * 'service_name.method_name' : Activates the debug logs for the specific method of the service, the variables won't be shown + * 'service_name.method_name.concept_id' : Activates the debug of a specific concept, within a specific method, for a specific service + * 'service_name.*' + + * path_to_concept when an integer is given + * concept_id + + * context_id + * context_id + * 'context_id+' (context_id followed by the sign '+') + + * debug_id + * debug_id \ No newline at end of file diff --git a/docs/source/blog/rules.rst b/docs/source/tech/rules.rst similarity index 51% rename from docs/source/blog/rules.rst rename to docs/source/tech/rules.rst index 5bd8b31..1f6c86c 100644 --- a/docs/source/blog/rules.rst +++ b/docs/source/tech/rules.rst @@ -1,6 +1,16 @@ Rules ======== +Abstract +**************** +As I previously explain, there are two main categories of object to make Sheerka come to life : + * the concepts + * the rules + +When the purpose of the concepts is to connect Sheerka with the outside world (our world), +The purpose of the rules is to define how to react in different situation. + +The combination of the two, the concepts and the rules, brings the intelligence. Basic definition @@ -10,15 +20,18 @@ To define a new rule :: - > when then + > when then|print Rules can have name, so you can also use the syntax :: - > def rule as when then + > def rule as when then|print +You can define action rule (using then) or display rule (using print). As of today, the two +sets of rules are different, but they may be merged if no notable difference is found. + Existing rule engines ********************* @@ -53,7 +66,7 @@ I am not an expert in rule engine. So I guess that the best way to figure out wh Use cases ********* -I see the rules engine like the caching service or the logging service. It can be used anywhere in the code. +I see the rules engine like the caching service or the logging service, in the way as it can be used anywhere in the code. It's not just a global feature of Sheerka. It's another way of achieving common task. For example, in the print service, I want to print all the failed ``ReturnValueConcept`` in red. @@ -92,9 +105,70 @@ In the predicate part, I need to control how expression are evaluated. In the predicate part, as well as in the action part, I must be able to used other concepts - -:: +For example, if I have the following concept already defined :: > def concept status is not ok as > def concept paint in red as + + +then I must be able to use :: + > when status is not ok then paint in red + + +Execution rules +*************** + +The first rule that I would like to define is the 'hello sheerka' rule. The principle is simple, if I enter 'hello sheerka', Sheerka should respond 'hello kodjo' + +First proposal :: + + > def greetings as hello x where x + > when greetings and greetings.x == Sheerka then 'hello kodjo' + + +Note that in this example, greetings is used name of concept when it is recognized, but also +as a name of variable in the test. + +Why not ? :: + + > def greetings as hello x where x + > when hello Sheerka then 'hello kodjo' + +which is different from :: + + > def greetings as hello x where x + > when 'hello Sheerka' then 'hello kodjo' + +In the first version, I supposed that the concept 'greetings' is recognized + +A more advanced version for the action part will be :: + + > when greetings and greetings.x == Sheerka then answer(call_concept('greetings', __user)) + +I said earlier that I would like the rule engine to be called whenever I want. So now, +there is the important question of when calling the rule. + +A rule like ``when greetings and greetings.x == Sheerka then 'hello kodjo'`` must be called after than +the concept ``greetings`` is recognized. But Should I call it before or after the 'Print' ? + +And should I consider the response as a new event, this time triggerd by Sheerka, rather than by a user ? + + +Implementation +*************** + +The 'implementation' deals with how the rules are evaluated. I have two engines. + +The first one, within the class ``SheerkaEvaluateRule`` is a simple and naive implementation. +It sorts the rule by priorities and evaluate them one after the other. Once rule matches the available data (the bag) +the algorithm stops. + +The other implementation is more sophisticated. It's an implementation of the 'Rete' +algorithm (https://en.wikipedia.org/wiki/Rete_algorithm). + +.. _phD: https://fr.wikipedia.org/wiki/Ulysse_31 +.. _PyRete: https://github.com/cmaclell/py_rete + +My work is based on PyRete_ wich it is itself a Python implementation of +the phD_ of Doorenbos (1995) \ No newline at end of file diff --git a/src/core/ast_helpers.py b/src/core/ast_helpers.py index c93e84f..56a3d6e 100644 --- a/src/core/ast_helpers.py +++ b/src/core/ast_helpers.py @@ -92,6 +92,15 @@ class UnreferencedVariablesVisitor(UnreferencedNamesVisitor): class NamesWithAttributesVisitor(ast.NodeVisitor): + """ + Looks for all atrtibutes for a given name + >>> ast_ = ast.parse("foo.bar.baz", "", mode="exec") + >>> assert NamesWithAttributesVisitor().get_sequences(ast_, "foo") == [["foo", "bar", "baz"]] + + It parses all expressions / statements + >>> ast_ = ast.parse("foo.bar.baz; one.two.three; foo.bar", "", mode="exec") + >>> assert NamesWithAttributesVisitor().get_sequences(ast_, "foo") == [["foo", "bar", "baz"], ["foo", "bar"]] + """ def __init__(self): self.sequences = [] diff --git a/src/core/builtin_concepts.py b/src/core/builtin_concepts.py index 98a0422..395a53f 100644 --- a/src/core/builtin_concepts.py +++ b/src/core/builtin_concepts.py @@ -1,6 +1,6 @@ from core.builtin_concepts_ids import BuiltinConcepts from core.concept import Concept, ConceptParts -from core.error import ErrorObj +from core.global_symbols import ErrorObj class UserInputConcept(Concept): @@ -167,6 +167,37 @@ class ParserResultConcept(Concept): return parser.name if isinstance(parser, BaseParser) else str(parser) +class RuleEvaluationResultConcept(Concept): + """ + Result of the evaluation of a rule, using the Rete algorithm + """ + + ALL_ATTRIBUTES = ["rule"] + + def __init__(self, rule=None, concept_id=None): + Concept.__init__(self, + BuiltinConcepts.RULE_EVALUATION_RESULT, + True, + False, + BuiltinConcepts.RULE_EVALUATION_RESULT, + id=concept_id, + bound_body="rule") + self.set_value("rule", rule) + self._metadata.is_evaluated = True + + def __repr__(self): + return f"RuleEvaluationResult(rule={self.rule})" + + def __eq__(self, other): + if not isinstance(other, RuleEvaluationResultConcept): + return False + + return self.rule == other.rule + + def __hash__(self): + return hash((self._metadata.name, self.rule)) + + class InvalidReturnValueConcept(Concept, ErrorObj): """ Error returned when an evaluator is not correctly coded diff --git a/src/core/builtin_concepts_ids.py b/src/core/builtin_concepts_ids.py index bbf622d..6f5c403 100644 --- a/src/core/builtin_concepts_ids.py +++ b/src/core/builtin_concepts_ids.py @@ -33,6 +33,9 @@ class BuiltinConcepts: BEFORE_RENDERING = "__BEFORE_RENDERING" # activate before the output is rendered RENDERING = "__RENDERING" # rendering the response from sheerka AFTER_RENDERING = "__AFTER_RENDERING" # rendering the response from sheerka + BEFORE_RULES_EVALUATION = "__BEFORE_RULES_EVALUATION" # just before evaluating rules + RULES_EVALUATION = "__RULES_EVALUATION" # evaluating rules + AFTER_RULES_EVALUATION = "__AFTER_RULES_EVALUATION" # after evaluating rules EVALUATE_SOURCE = "__EVALUATE_SOURCE" # EVALUATE_CONCEPT = "__EVALUATE_CONCEPT" # a concept will be evaluated EVALUATING_CONCEPT = "__EVALUATING_CONCEPT" # a concept will be evaluated @@ -46,12 +49,12 @@ class BuiltinConcepts: EXEC_CODE = "__EXEC_CODE" # to use when executing Python or other language compiled code TESTING = "__TESTING" EVALUATOR_PRE_PROCESS = "__EVALUATOR_PRE_PROCESS" # used modify / tweak behaviour of evaluators - EVALUATING_RULES = "__EVALUATING_RULES" # builtin attributes ISA = "__ISA" # when a concept is an instance of another one HASA = "__HASA" # when a concept has/owns another concept AUTO_EVAL = "__AUTO_EVAL" # when the concept must be auto evaluated + RECOGNIZED_BY = "__RECOGNIZED_BY" # indicate how a concept was recognized # object USER_INPUT = "__USER_INPUT" # represent an input from an user @@ -64,6 +67,7 @@ class BuiltinConcepts: NEW_CONCEPT = "__NEW_CONCEPT" # when a new concept is added UNKNOWN_PROPERTY = "__UNKNOWN_PROPERTY" # when requesting for a unknown property PARSER_RESULT = "__PARSER_RESULT" + RULE_EVALUATION_RESULT = "__RULE_EVALUATION_RESULT" TOO_MANY_SUCCESS = "__TOO_MANY_SUCCESS" # when expecting a limited number of successful return value TOO_MANY_ERRORS = "__TOO_MANY_ERRORS" # when expecting a limited number of successful return value ONLY_SUCCESSFUL = "__ONLY_SUCCESSFUL" # filter the result, only keep successful ones diff --git a/src/core/builtin_helpers.py b/src/core/builtin_helpers.py index 02de8fb..d886f07 100644 --- a/src/core/builtin_helpers.py +++ b/src/core/builtin_helpers.py @@ -13,6 +13,7 @@ from core.utils import as_bag from parsers.BaseNodeParser import SourceCodeNode, ConceptNode, UnrecognizedTokensNode, SourceCodeWithConceptNode, \ RuleNode from parsers.BaseParser import ParsingError +from parsers.PythonParser import PythonParser PARSE_STEPS = [BuiltinConcepts.BEFORE_PARSING, BuiltinConcepts.PARSING, BuiltinConcepts.AFTER_PARSING] EVAL_STEPS = PARSE_STEPS + [BuiltinConcepts.BEFORE_EVALUATION, BuiltinConcepts.EVALUATION, @@ -235,7 +236,7 @@ def only_parsers_results(context, return_values): Filters the return_values and returns when the result is a ParserResult regardless of the status - So it filters errors + So it filters parsers in error (ERROR, NOT_FOR_ME, EMPTY...) :param context: :param return_values: :return: @@ -332,7 +333,7 @@ def parse_unrecognized(context, source, parsers, who=None, prop=None, filter_fun def parse_function(context, source, tokens=None, start=0): """ - Helper function to parse what is supposed to be a function + Helper function that parses what is supposed to be a function :param context: :param source: :param tokens: @@ -361,6 +362,34 @@ def parse_function(context, source, tokens=None, start=0): return res +def parse_python(context, source, desc=None): + """ + Helper function that parses what is known to be Python source code + :param context: + :param source: + :param desc: option description when creating the sub context + """ + desc = desc or f"Compiling python '{source}'" + with context.push(BuiltinConcepts.PARSE_CODE, + {"language": "Python", "source": source}, + desc) as sub_context: + parser_input = context.sheerka.services[SheerkaExecute.NAME].get_parser_input(source) + python_parser = PythonParser() + return python_parser.parse(sub_context, parser_input) + + +def parse_expression(context, source, desc=None): + """ + Helper function to parser expressions with AND, OR and NOT + """ + desc = desc or f"Parsing expression '{source}'" + with context.push(BuiltinConcepts.PARSE_CODE, source, desc) as sub_context: + parser_input = context.sheerka.services[SheerkaExecute.NAME].get_parser_input(source) + from parsers.ExpressionParser import ExpressionParser + expr_parser = ExpressionParser() + return expr_parser.parse(sub_context, parser_input) + + def evaluate(context, source, evaluators="all", @@ -472,12 +501,67 @@ def get_lexer_nodes(return_values, start, tokens): return lexer_nodes -def ensure_evaluated(context, concept, eval_body=True): +def get_lexer_nodes_using_positions(return_values, positions): + """ + Transform all elements from return_values into lexer nodes + use positions to remap the exact positions + """ + lexer_nodes = [] + for ret_val, position in zip(return_values, positions): + if ret_val.who in ("parsers.Python", 'parsers.PythonWithConcepts'): + + lexer_nodes.append(SourceCodeNode(position.start, + position.end, + position.tokens, + ret_val.body.source, + python_node=ret_val.body.body, + return_value=ret_val)) + + elif ret_val.who == "parsers.ExactConcept": + concepts = ret_val.body.body if hasattr(ret_val.body.body, "__iter__") else [ret_val.body.body] + for concept in concepts: + lexer_nodes.append(ConceptNode(concept, + position.start, + position.end, + position.tokens, + ret_val.body.source)) + + elif ret_val.who in ("parsers.Bnf", "parsers.Sya", "parsers.Sequence"): + nodes = [node for node in ret_val.body.body] + for node in nodes: + node.start = position.start + node.end = position.end + + # but append the whole sequence if when it's a sequence + lexer_nodes.extend(nodes) + + elif ret_val.who == "parsers.Rule": + rules = ret_val.body.body if hasattr(ret_val.body.body, "__iter__") else [ret_val.body.body] + for rule in rules: + lexer_nodes.append(RuleNode(rule, + position.start, + position.end, + position.tokens, ret_val.body.source)) + + elif ret_val.who == "parsers.Function": + node = ret_val.body.body + node.start = position.start + node.end = position.end + lexer_nodes.append(node) + + else: + raise NotImplementedError() + + return lexer_nodes + + +def ensure_evaluated(context, concept, eval_body=True, metadata=None): """ Evaluate a concept is not already evaluated :param context: :param concept: :param eval_body: + :param metadata: :return: """ if concept.get_metadata().is_evaluated: @@ -485,13 +569,13 @@ def ensure_evaluated(context, concept, eval_body=True): # do not try to evaluate concept that are not fully initialized if concept.get_metadata().definition_type != DEFINITION_TYPE_BNF: - for var in concept.get_metadata().variables: - if var[1] is None and \ - var[0] not in concept.get_compiled() and \ - (var[0] not in concept.values() or concept.get_value(var[0]) == NotInit): + for var_name, var_default_value in concept.get_metadata().variables: + if var_default_value is None and \ + var_name not in concept.get_compiled() and \ + (var_name not in concept.values() or concept.get_value(var_name) == NotInit): return concept - evaluated = context.sheerka.evaluate_concept(context, concept, eval_body=eval_body) + evaluated = context.sheerka.evaluate_concept(context, concept, eval_body=eval_body, metadata=metadata) return evaluated @@ -731,3 +815,90 @@ def evaluate_object(bag, properties): bag = as_bag(obj) return obj + + +def is_a_question(context, concept): + """ + Returns True if the concept must be executed in the context of BuiltinConcepts.EVAL_QUESTION_REQUESTED + The only two ways that are currently supported are + * is_question() appears in the pre condition + * context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED) appears in the pre condition + :param context: + :param concept: concept to analyse + """ + pre = concept.get_metadata().pre + if pre in (None, NotInit, ""): + return False + + parser_input_service = context.sheerka.services[SheerkaExecute.NAME] + from parsers.ExpressionParser import ExpressionParser + parser = ExpressionParser() + + res = parser.parse(context, parser_input_service.get_parser_input(pre)) + if not res.status: + return False + + node = res.body.body + from parsers.expressions import IsAQuestionVisitor + return IsAQuestionVisitor().is_a_question(node) + + +def get_inner_body(context, concept): + """ + For container concept, returns the body + """ + if context.sheerka.isinstance(concept.body, BuiltinConcepts.ONLY_SUCCESSFUL): + return concept.body.body + else: + return concept.body + + +class CreateObjectIdentifiers: + """ + Class that creates unique identifiers for Concept or Rule objects + """ + + def __init__(self): + self.identifiers = {} + self.identifiers_key = {} + + @staticmethod + def sanitize(identifier): + if identifier is None: + return "" + + res = "" + for c in identifier: + res += c if c.isalnum() else "0" + return res + + def get_identifier(self, obj, wrapper): + """ + Get an identifier for a concept. + Make sure to return the same identifier if the same concept + Make sure to return a different identifier if same name but different concept + + Internal function because I don't want identifiers, identifiers_key and python_ids_mappings + to be instance variables + I would like to keep this parser as stateless as possible + :param obj: + :param wrapper: string or char that will wrap the result (ex '__C__' or '__R__') + :return: + """ + if id(obj) in self.identifiers: + return self.identifiers[id(obj)] + + identifier = wrapper + self.sanitize(obj.key or obj.name) + if obj.id: + identifier += "__" + obj.id + + if identifier in self.identifiers_key: + self.identifiers_key[identifier] += 1 + identifier += f"_{self.identifiers_key[identifier]}" + else: + self.identifiers_key[identifier] = 0 + + identifier += wrapper + + self.identifiers[id(obj)] = identifier + return identifier diff --git a/src/core/concept.py b/src/core/concept.py index b58f5e6..8ba206b 100644 --- a/src/core/concept.py +++ b/src/core/concept.py @@ -150,6 +150,7 @@ class Concept: self._bnf = None # parsing expression self._original_definition_hash = None # concept hash before any alteration of the metadata self._format = None # how to print the concept + self._hints = {} # extra processing information to help processing def __repr__(self): text = f"({self._metadata.id}){self._metadata.name}" @@ -499,6 +500,15 @@ class Concept: return {k: v for k, v in self.values().items() if not k[0] == "#"} # return dict([(k, v) for k, v in self.values.items() if isinstance(k, str)]) + def set_hint(self, name, value): + self._hints[name] = value + + def get_hint(self, name): + try: + return self._hints[name] + except KeyError: + return None + def auto_init(self): """ Sometimes (for tests purposes) @@ -533,10 +543,9 @@ class Concept: It quicker to implement than creating the actual property mechanism with @property And it removes the visibility from the other attributes/methods """ - bag = self.variables() - - for prop in ("id", "name", "key", "body"): - bag[prop] = getattr(self, prop) + bag = {prop: getattr(self, prop) for prop in ("id", "name", "key")} + bag.update(self.variables()) + bag["body"] = getattr(self, "body") return bag def as_debug_bag(self, new_obj, recurse): diff --git a/src/core/error.py b/src/core/error.py deleted file mode 100644 index a1b0489..0000000 --- a/src/core/error.py +++ /dev/null @@ -1,6 +0,0 @@ -class ErrorObj: - """ - To indicate that somehow, the underlying object is (or has) an error - """ - pass - diff --git a/src/core/global_symbols.py b/src/core/global_symbols.py index df303ab..c7e2ce2 100644 --- a/src/core/global_symbols.py +++ b/src/core/global_symbols.py @@ -1,9 +1,17 @@ # events -EVENT_CONCEPT_PRECEDENCE_MODIFIED = "evt_cpm" -EVENT_RULE_PRECEDENCE_MODIFIED = "evt_rpm" -EVENT_CONTEXT_DISPOSED = "evt_cd" -EVENT_USER_INPUT_EVALUATED = "evt_uie" -EVENT_CONCEPT_CREATED = "evt_cc" + +EVENT_CONCEPT_PRECEDENCE_MODIFIED = "evt_cp_m" +EVENT_RULE_PRECEDENCE_MODIFIED = "evt_rp_m" +EVENT_CONTEXT_DISPOSED = "evt_ctx_d" +EVENT_USER_INPUT_EVALUATED = "evt_ui_e" +EVENT_CONCEPT_CREATED = "evt_c_c" +EVENT_CONCEPT_DELETED = "evt_c_d" +EVENT_CONCEPT_ID_DELETED = "evt_c_id_d" +EVENT_RULE_CREATED = "evt_r_c" +EVENT_RULE_DELETED = "evt_r_d" +EVENT_RULE_ID_DELETED = "evt_r_id_d" +EVENT_ONTOLOGY_CREATED = "evt_o_c" +EVENT_ONTOLOGY_DELETED = "evt_o_d" # comparison context RULE_COMPARISON_CONTEXT = "Rule" @@ -40,3 +48,10 @@ class RemovedType(CustomType): NotInit = NotInitType() NotFound = NotFoundType() Removed = RemovedType() + + +class ErrorObj: + """ + To indicate that somehow, the underlying object is (or has) an error + """ + pass diff --git a/src/core/rule.py b/src/core/rule.py index 84d9d89..9516c17 100644 --- a/src/core/rule.py +++ b/src/core/rule.py @@ -5,6 +5,7 @@ import core.utils ACTION_TYPE_PRINT = "print" ACTION_TYPE_EXEC = "exec" +ACTION_TYPE_TEST = "test" @dataclass @@ -30,17 +31,23 @@ class Rule: rule_id=None, is_enabled=None): self.metadata = RuleMetadata(action_type, name, predicate, action, id=rule_id, is_enabled=is_enabled) - self.compiled_predicate = None + self.compiled_predicates = None self.compiled_action = None from core.sheerka.services.SheerkaComparisonManager import SheerkaComparisonManager self.priority = priority if priority is not None else SheerkaComparisonManager.DEFAULT_COMPARISON_VALUE self.error_sink = None + # from SheerkaRete, not quite sure one when it will be used + self.rete_net = None + self.rete_p_nodes = [] # list of production nodes for this rule + self.rete_disjunctions = None # list of list as it may be several interpretation for a rule + def __repr__(self): rule_id = f"#{self.metadata.id}" if self.name: rule_id += f" ({self.metadata.name})" - return f"Rule({rule_id}, when '{self.metadata.predicate}' {self.metadata.action_type} '{self.metadata.action}', priority={self.priority})" + action_type = "print" if self.metadata.action_type == "print" else "then" + return f"Rule({rule_id}, when '{self.metadata.predicate}' {action_type} '{self.metadata.action}', priority={self.priority})" def __eq__(self, other): if id(other) == id(self): @@ -69,8 +76,12 @@ class Rule: self.priority, self.id, self.metadata.is_enabled) - copy.compiled_predicate = self.compiled_predicate + + copy.compiled_predicates = self.compiled_predicates copy.compiled_action = self.compiled_action + copy.metadata.is_compiled = self.metadata.is_compiled + copy.metadata.id_is_unresolved = self.metadata.id_is_unresolved + # copy.error_sink = self.error_sink # Uncomment this line if necessary return copy @@ -102,3 +113,6 @@ class Rule: def short_str(self): return f"Rule(#{self.metadata.id}, '{self.metadata.predicate}', priority={self.priority})" + + def get_rete_disjunctions(self): + return self.rete_disjunctions diff --git a/src/core/sheerka/ExecutionContext.py b/src/core/sheerka/ExecutionContext.py index ae43241..be6617b 100644 --- a/src/core/sheerka/ExecutionContext.py +++ b/src/core/sheerka/ExecutionContext.py @@ -1,5 +1,4 @@ import logging -import os import pprint import time @@ -87,9 +86,6 @@ class ExecutionContext: self.obj = obj self.concepts = concepts - self_debug, self.debug_mode = sheerka.get_context_debug_mode(self.id) - self.debug_enabled = self_debug is not None - @property def elapsed(self): if self._start == 0: @@ -184,10 +180,6 @@ class ExecutionContext: new.preprocess_evaluators = self.preprocess_evaluators new.protected_hints.update(self.protected_hints) - if new.debug_mode is None and self.debug_mode == "protected": - new.debug_mode = "protected" - new.debug_enabled = True - self._children.append(new) return new @@ -315,9 +307,10 @@ class ExecutionContext: to_str = self.return_value_to_str(r) self._logger.debug(f"[{self._id:2}]" + self._tab + "-> " + to_str) - def get_debugger(self, who, method_name): - return self.sheerka.get_debugger(self, who, method_name) + def get_debugger(self, who, method_name, new_debug_id=True): + return self.sheerka.get_debugger(self, who, method_name, new_debug_id) + # TODO: TO REMOVE def debug(self, who, method_name, variable_name, text, is_error=False): activated = self.sheerka.debug_activated_for(who) if activated: @@ -330,6 +323,7 @@ class ExecutionContext: self.sheerka.debug(f"[{self._id:3}] {CCM[color]}{who}.{method_name}.{variable_name}: {CCM['reset']}") self.sheerka.debug(str_text) + # TODO: TO REMOVE def debug_entering(self, who, method_name, **kwargs): if self.sheerka.debug_activated_for(who): str_text = pp.pformat(kwargs) @@ -340,6 +334,7 @@ class ExecutionContext: self.sheerka.debug(f"[{self._id:3}] {CCM['blue']}Entering {who}.{method_name}:{CCM['reset']}") self.sheerka.debug(f"[{self._id:3}] {str_text}") + # TODO: TO REMOVE def debug_log(self, who, text): if self.sheerka.debug_activated_for(who): self.sheerka.debug(f"[{self._id:3}] {CCM['blue']}{text}{CCM['reset']}") @@ -493,5 +488,3 @@ class ExecutionContext: return False return False - - diff --git a/src/core/sheerka/Sheerka.py b/src/core/sheerka/Sheerka.py index e3b86a5..7d15e0d 100644 --- a/src/core/sheerka/Sheerka.py +++ b/src/core/sheerka/Sheerka.py @@ -1,6 +1,7 @@ import inspect import logging from dataclasses import dataclass +from operator import attrgetter import core.builtin_helpers import core.utils @@ -10,8 +11,7 @@ from cache.IncCache import IncCache from core.builtin_concepts import ErrorConcept, ReturnValueConcept, UnknownConcept from core.builtin_concepts_ids import BuiltinErrors, BuiltinConcepts from core.concept import Concept, ConceptParts, get_concept_attrs -from core.error import ErrorObj -from core.global_symbols import EVENT_USER_INPUT_EVALUATED, NotInit, NotFound +from core.global_symbols import EVENT_USER_INPUT_EVALUATED, NotInit, NotFound, ErrorObj, EVENT_ONTOLOGY_CREATED from core.profiling import profile from core.sheerka.ExecutionContext import ExecutionContext from core.sheerka.SheerkaOntologyManager import SheerkaOntologyManager, OntologyAlreadyExists @@ -31,6 +31,23 @@ EXECUTE_STEPS = [ BuiltinConcepts.AFTER_EVALUATION ] +RULES_EVALUATE_STEPS = [ + BuiltinConcepts.BEFORE_RULES_EVALUATION, + BuiltinConcepts.RULES_EVALUATION, + BuiltinConcepts.AFTER_RULES_EVALUATION, +] + +RULES_EXECUTE_STEPS = [ + BuiltinConcepts.BEFORE_EVALUATION, + BuiltinConcepts.EVALUATION, + BuiltinConcepts.AFTER_EVALUATION +] + +# when a concept is instantiated via resolve or false_resolve +# It indicate which parameter was used to recognize the concept +RECOGNIZED_BY_ID = "by_id" +RECOGNIZED_BY_NAME = "by_name" + @dataclass class SheerkaMethod: @@ -92,17 +109,25 @@ class Sheerka(Concept): self.save_execution_context = True self.enable_process_return_values = True + self.enable_process_rules = True self.methods_with_context = {"test_using_context"} # only the names, the method is defined in sheerka_methods self.sheerka_methods = { "test": SheerkaMethod(self.test, False), "test_using_context": SheerkaMethod(self.test_using_context, False), - "test_dict": SheerkaMethod(self.test_dict, False) + "test_dict": SheerkaMethod(self.test_dict, False), + "test_error": SheerkaMethod(self.test_error, False), } self.locals = {} self.concepts_ids = None + def __copy__(self): + return self + + def __deepcopy__(self, memodict={}): + return self + @property def concepts_grammars(self): """ @@ -138,7 +163,7 @@ class Sheerka(Concept): setattr(self, bound_method.__name__, bound_method) - def initialize(self, root_folder: str = None, save_execution_context=None, enable_process_return_values=None): + def initialize(self, root_folder: str = None, **kwargs): """ Starting Sheerka Loads the current configuration @@ -149,11 +174,10 @@ class Sheerka(Concept): :return: ReturnValue(Success or Error) """ - if save_execution_context is not None: - self.save_execution_context = save_execution_context - - if enable_process_return_values is not None: - self.enable_process_return_values = enable_process_return_values + self.save_execution_context = kwargs.get("save_execution_context", self.save_execution_context) + self.enable_process_return_values = kwargs.get("enable_process_return_values", + self.enable_process_return_values) + self.enable_process_rules = kwargs.get("enable_process_rules", self.enable_process_rules) try: self.during_initialisation = True @@ -168,6 +192,7 @@ class Sheerka(Concept): self.get_builtin_evaluators() self.initialize_services() self.initialize_builtin_evaluators() + self.om.init_subscribers() event = Event("Initializing Sheerka.", user_id=self.name) self.om.save_event(event) @@ -234,11 +259,12 @@ class Sheerka(Concept): core.utils.import_module_and_sub_module('core.sheerka.services') base_class = "core.sheerka.services.sheerka_service.BaseService" - for service in core.utils.get_sub_classes("core.sheerka.services", base_class): - instance = service(self) - if hasattr(instance, "initialize"): - instance.initialize() - self.services[service.NAME] = instance + services = [service(self) for service in core.utils.get_sub_classes("core.sheerka.services", base_class)] + services.sort(key=attrgetter("order")) + for service in services: + if hasattr(service, "initialize"): + service.initialize() + self.services[service.NAME] = service def initialize_services_deferred(self, context, is_first_time): """ @@ -325,7 +351,6 @@ class Sheerka(Concept): ontologies = self.om.current_sdp().load_ontologies() if not ontologies: return - for ontology_name in list(reversed(ontologies))[1:]: self.om.push_ontology(ontology_name, False) self.initialize_services_deferred(context, False) @@ -360,6 +385,10 @@ class Sheerka(Concept): ret = self.execute(execution_context, [user_input, reduce_requested], EXECUTE_STEPS) execution_context.add_values(return_values=ret) + # rule management + if self.enable_process_rules: + ret = self.execute_rules(execution_context, ret, RULES_EVALUATE_STEPS, RULES_EXECUTE_STEPS) + if self.om.is_dirty: self.om.commit(execution_context) @@ -388,10 +417,14 @@ class Sheerka(Concept): :return: """ - def new_instances(concepts): + def add_recognized_by(c, _recognized_by): + c.set_hint(BuiltinConcepts.RECOGNIZED_BY, _recognized_by) + return c + + def new_instances(concepts, _recognized_by): if hasattr(concepts, "__iter__"): - return [self.new_from_template(c, c.key) for c in concepts] - return self.new_from_template(concepts, concepts.key) + return [add_recognized_by(self.new_from_template(c, c.key), _recognized_by) for c in concepts] + return add_recognized_by(self.new_from_template(concepts, concepts.key), _recognized_by) if concept is None: return None @@ -421,10 +454,11 @@ class Sheerka(Concept): if self.is_known(found := self.get_by_id(concept[1])): instance = self.new_from_template(found, found.key) instance._metadata.is_evaluated = True + instance.set_hint(BuiltinConcepts.RECOGNIZED_BY, RECOGNIZED_BY_ID) return instance elif concept[0]: if self.is_known(found := self.get_by_name(concept[0])): - instances = new_instances(found) + instances = new_instances(found, RECOGNIZED_BY_NAME) core.builtin_helpers.set_is_evaluated(instances) return instances else: @@ -433,17 +467,22 @@ class Sheerka(Concept): # otherwise search in db if isinstance(concept, str): if self.is_known(found := self.get_by_name(concept)): - instances = new_instances(found) + instances = new_instances(found, RECOGNIZED_BY_NAME) core.builtin_helpers.set_is_evaluated(instances, check_nb_variables=True) return instances return None def fast_resolve(self, key, return_new=True): - def new_instances(concepts): + def add_recognized_by(c, _recognized_by): + c.set_hint(BuiltinConcepts.RECOGNIZED_BY, _recognized_by) + return c + + def new_instances(concepts, _recognized_by): if hasattr(concepts, "__iter__"): - return [self.new_from_template(c, c.key) for c in concepts] - return self.new_from_template(concepts, concepts.key) + return [add_recognized_by(self.new_from_template(c, c.key), _recognized_by) for c in concepts] + + return add_recognized_by(self.new_from_template(concepts, concepts.key), _recognized_by) if isinstance(key, Token): if key.type == TokenKind.RULE: # do not recognize rules !!! @@ -459,14 +498,17 @@ class Sheerka(Concept): if key[1]: concept = self.om.get(self.CONCEPTS_BY_ID_ENTRY, key[1]) + recognized_by = RECOGNIZED_BY_ID else: concept = self.om.get(self.CONCEPTS_BY_NAME_ENTRY, key[0]) + recognized_by = RECOGNIZED_BY_NAME else: concept = self.om.get(self.CONCEPTS_BY_NAME_ENTRY, key) + recognized_by = RECOGNIZED_BY_NAME if concept is NotFound: return None - return new_instances(concept) if return_new else concept + return new_instances(concept, recognized_by) if return_new else concept def new(self, concept_key, **kwargs): """ @@ -478,6 +520,8 @@ class Sheerka(Concept): """ if isinstance(concept_key, tuple): concept_key, concept_id = concept_key[0], concept_key[1] + elif isinstance(concept_key, Concept): + concept_key, concept_id = concept_key.key, concept_key.id else: concept_id = None @@ -547,12 +591,13 @@ class Sheerka(Concept): if name in self.om.current_sdp().load_ontologies(): self.initialize_services_deferred(context, False) - self.om.save_ontologies() + self.om.save_ontologies_names() + self.publish(context, EVENT_ONTOLOGY_CREATED, name) return self.ret(self.name, True, self.new(BuiltinConcepts.SUCCESS)) - def pop_ontology(self): - ontology = self.om.pop_ontology() + def pop_ontology(self, context): + ontology = self.om.pop_ontology(context) self.om.reset_sheerka_state() for service in self.services.values(): @@ -561,7 +606,7 @@ class Sheerka(Concept): if hasattr(service, "reset_state"): service.reset_state() - self.om.save_ontologies() + self.om.save_ontologies_names() return self.ret(self.name, True, self.new(BuiltinConcepts.ONTOLOGY_REMOVED, body=ontology)) def get_ontology(self, context): @@ -699,6 +744,13 @@ class Sheerka(Concept): return bool(obj) + @staticmethod + def is_error(obj): + """ + opposite of is_success + """ + return not Sheerka.is_success(obj) + @staticmethod def is_known(obj): if not isinstance(obj, Concept): diff --git a/src/core/sheerka/SheerkaOntologyManager.py b/src/core/sheerka/SheerkaOntologyManager.py index 8927f3a..50a4e5e 100644 --- a/src/core/sheerka/SheerkaOntologyManager.py +++ b/src/core/sheerka/SheerkaOntologyManager.py @@ -1,7 +1,10 @@ +from cache.Cache import Cache from cache.CacheManager import CacheManager from cache.DictionaryCache import DictionaryCache +from cache.SetCache import SetCache from core.concept import copy_concepts_attrs, load_concepts_attrs -from core.global_symbols import NotFound, Removed +from core.global_symbols import NotFound, Removed, EVENT_CONCEPT_CREATED, EVENT_CONCEPT_DELETED, EVENT_RULE_CREATED, \ + EVENT_RULE_DELETED, EVENT_CONCEPT_ID_DELETED, EVENT_RULE_ID_DELETED from core.utils import sheerka_deepcopy from sdp.sheerkaDataProvider import SheerkaDataProvider @@ -88,6 +91,11 @@ class Ontology: class SheerkaOntologyManager: ROOT_ONTOLOGY_NAME = "__default__" + SELF_CACHE_MANAGER = "__ontology_manager__" # cache to store SheerkaOntologyManager info + CONCEPTS_BY_ONTOLOGY_ENTRY = "ConceptsByOntologyEntry" + RULES_BY_ONTOLOGY_ENTRY = "RulesByOntologyEntry" + ONTOLOGY_BY_CONCEPT_ENTRY = "OntologyByConceptEntry" + ONTOLOGY_BY_RULE_ENTRY = "OntologyByRuleEntry" def __init__(self, sheerka, root_folder, cache_only): self.sheerka = sheerka @@ -98,6 +106,20 @@ class SheerkaOntologyManager: ref_cache_manager = CacheManager(self.cache_only, sdp=SheerkaDataProvider(root_folder, self.sheerka)) self.ontologies = [Ontology(self.ROOT_ONTOLOGY_NAME, ref_cache_manager, None)] + self_sdp = SheerkaDataProvider(root_folder, self.sheerka, self.SELF_CACHE_MANAGER) + self.self_cache_manager = CacheManager(self.cache_only, sdp=self_sdp) + cache = SetCache(max_size=None).auto_configure(self.CONCEPTS_BY_ONTOLOGY_ENTRY) + self.self_cache_manager.register_cache(self.CONCEPTS_BY_ONTOLOGY_ENTRY, cache) + + cache = SetCache(max_size=None).auto_configure(self.RULES_BY_ONTOLOGY_ENTRY) + self.self_cache_manager.register_cache(self.RULES_BY_ONTOLOGY_ENTRY, cache) + + cache = Cache(max_size=None).auto_configure(self.ONTOLOGY_BY_CONCEPT_ENTRY) + self.self_cache_manager.register_cache(self.ONTOLOGY_BY_CONCEPT_ENTRY, cache) + + cache = Cache(max_size=None).auto_configure(self.ONTOLOGY_BY_RULE_ENTRY) + self.self_cache_manager.register_cache(self.ONTOLOGY_BY_RULE_ENTRY, cache) + @property def ontologies_names(self): return [o.name for o in self.ontologies] @@ -111,6 +133,12 @@ class SheerkaOntologyManager: self.frozen = False return self + def init_subscribers(self): + self.sheerka.subscribe(EVENT_CONCEPT_CREATED, self.on_concept_created) + self.sheerka.subscribe(EVENT_CONCEPT_DELETED, self.on_concept_deleted) + self.sheerka.subscribe(EVENT_RULE_CREATED, self.on_rule_created) + self.sheerka.subscribe(EVENT_RULE_DELETED, self.on_rule_deleted) + def push_ontology(self, name, cache_only=None): """ Add an ontology layer @@ -138,7 +166,7 @@ class SheerkaOntologyManager: self.ontologies.insert(0, Ontology(name, cache_manager, alt_sdp)) return self - def pop_ontology(self): + def pop_ontology(self, context): """ Remove the top ontology layer """ @@ -148,6 +176,22 @@ class SheerkaOntologyManager: if len(self.ontologies) == 1: raise OntologyManagerCannotPopLatest() + # remove concepts and rules tracking for the ontology to pop + ontology_name = self.current_ontology().name + concepts = self.self_cache_manager.get(self.CONCEPTS_BY_ONTOLOGY_ENTRY, ontology_name) + if concepts is not NotFound: + for concept in concepts: + self.sheerka.publish(context, EVENT_CONCEPT_ID_DELETED, concept) + self.self_cache_manager.delete(self.ONTOLOGY_BY_CONCEPT_ENTRY, concept) + self.self_cache_manager.delete(self.CONCEPTS_BY_ONTOLOGY_ENTRY, ontology_name) + + rules = self.self_cache_manager.get(self.RULES_BY_ONTOLOGY_ENTRY, ontology_name) + if rules is not NotFound: + for rule in rules: + self.sheerka.publish(context, EVENT_RULE_ID_DELETED, rule) + self.self_cache_manager.delete(self.ONTOLOGY_BY_RULE_ENTRY, rule) + self.self_cache_manager.delete(self.RULES_BY_ONTOLOGY_ENTRY, ontology_name) + return self.ontologies.pop(0) def add_ontology(self, ontology: Ontology): @@ -179,7 +223,7 @@ class SheerkaOntologyManager: raise KeyError(name) - def save_ontologies(self): + def save_ontologies_names(self): self.current_sdp().save_ontologies(self.ontologies_names) # def load_ontologies(self): @@ -446,6 +490,7 @@ class SheerkaOntologyManager: :param context: :return: """ + self.self_cache_manager.commit(context) return self.current_cache_manager().commit(context) def clear(self, cache_name=None): @@ -468,3 +513,21 @@ class SheerkaOntologyManager: def is_dirty(self): return self.current_cache_manager().is_dirty + + def on_concept_created(self, context, concept): + self.self_cache_manager.put(self.CONCEPTS_BY_ONTOLOGY_ENTRY, self.current_ontology().name, concept.id) + self.self_cache_manager.put(self.ONTOLOGY_BY_CONCEPT_ENTRY, concept.id, self.current_ontology().name) + + def on_concept_deleted(self, context, concept): + ontology_name = self.self_cache_manager.get(self.ONTOLOGY_BY_CONCEPT_ENTRY, concept.id) + self.self_cache_manager.delete(self.CONCEPTS_BY_ONTOLOGY_ENTRY, ontology_name, concept.id) + self.self_cache_manager.delete(self.ONTOLOGY_BY_CONCEPT_ENTRY, concept.id) + + def on_rule_created(self, context, rule): + self.self_cache_manager.put(self.RULES_BY_ONTOLOGY_ENTRY, self.current_ontology().name, rule.id) + self.self_cache_manager.put(self.ONTOLOGY_BY_RULE_ENTRY, rule.id, self.current_ontology().name) + + def on_rule_deleted(self, context, rule): + ontology_name = self.self_cache_manager.get(self.ONTOLOGY_BY_RULE_ENTRY, rule.id) + self.self_cache_manager.delete(self.RULES_BY_ONTOLOGY_ENTRY, ontology_name, rule.id) + self.self_cache_manager.delete(self.ONTOLOGY_BY_RULE_ENTRY, rule.id) diff --git a/src/core/sheerka/services/SheerkaAdmin.py b/src/core/sheerka/services/SheerkaAdmin.py index b5edcdf..5c8812c 100644 --- a/src/core/sheerka/services/SheerkaAdmin.py +++ b/src/core/sheerka/services/SheerkaAdmin.py @@ -3,7 +3,7 @@ import time from os import path from core.builtin_concepts_ids import BuiltinConcepts, BuiltinContainers -from core.builtin_helpers import ensure_concept +from core.builtin_helpers import ensure_concept_or_rule from core.concept import Concept from core.sheerka.services.SheerkaMemory import SheerkaMemory from core.sheerka.services.sheerka_service import BaseService @@ -28,6 +28,7 @@ class SheerkaAdmin(BaseService): self.sheerka.bind_service_method(self.extended_isinstance, False) self.sheerka.bind_service_method(self.is_container, False) self.sheerka.bind_service_method(self.format_rules, False) + self.sheerka.bind_service_method(self.exec_rules, False) self.sheerka.bind_service_method(self.admin_push_ontology, True, as_name="push_ontology") self.sheerka.bind_service_method(self.admin_pop_ontology, True, as_name="pop_ontology") self.sheerka.bind_service_method(self.ontologies, False) @@ -127,24 +128,36 @@ class SheerkaAdmin(BaseService): concepts = sorted(self.sheerka.om.list(self.sheerka.CONCEPTS_BY_ID_ENTRY), key=lambda item: int(item.id)) return self.sheerka.new(BuiltinConcepts.TO_LIST, body=concepts) - def desc(self, *concepts): - ensure_concept(*concepts) + def desc(self, *items): + ensure_concept_or_rule(*items) res = [] - for c in concepts: - bag = { - "id": c.id, - "name": c.name, - "key": c.key, - "definition": c.get_metadata().definition, - "type": c.get_metadata().definition_type, - "body": c.get_metadata().body, - "where": c.get_metadata().where, - "pre": c.get_metadata().pre, - "post": c.get_metadata().post, - "ret": c.get_metadata().ret, - "vars": c.get_metadata().variables, - "props": c.get_metadata().props, - } + for item in items: + if isinstance(item, Concept): + bag = { + "id": item.id, + "name": item.name, + "key": item.key, + "definition": item.get_metadata().definition, + "type": item.get_metadata().definition_type, + "body": item.get_metadata().body, + "where": item.get_metadata().where, + "pre": item.get_metadata().pre, + "post": item.get_metadata().post, + "ret": item.get_metadata().ret, + "vars": item.get_metadata().variables, + "props": item.get_metadata().props, + } + else: + bag = { + "id": item.id, + "name": item.metadata.name, + "type": item.metadata.action_type, + "predicate": item.metadata.predicate, + "action": item.metadata.action, + "priority": item.priority, + "compiled": item.metadata.is_compiled, + "enabled": item.metadata.is_enabled, + } res.append(self.sheerka.new(BuiltinConcepts.TO_DICT, body=bag)) return res[0] if len(res) == 1 else self.sheerka.new(BuiltinConcepts.TO_LIST, body=res) @@ -152,6 +165,9 @@ class SheerkaAdmin(BaseService): def format_rules(self): return self.sheerka.new(BuiltinConcepts.TO_LIST, items=self.sheerka.get_format_rules()) + def exec_rules(self): + return self.sheerka.new(BuiltinConcepts.TO_LIST, items=self.sheerka.get_exec_rules()) + def extended_isinstance(self, a, b): """ switch between sheerka.isinstance and builtin.isinstance @@ -180,8 +196,8 @@ class SheerkaAdmin(BaseService): def admin_push_ontology(self, context, name): return self.sheerka.push_ontology(context, name, False) - def admin_pop_ontology(self): - return self.sheerka.pop_ontology() + def admin_pop_ontology(self, context): + return self.sheerka.pop_ontology(context) def ontologies(self): ontologies = self.sheerka.om.ontologies_names diff --git a/src/core/sheerka/services/SheerkaComparisonManager.py b/src/core/sheerka/services/SheerkaComparisonManager.py index 6cfbc04..568e4eb 100644 --- a/src/core/sheerka/services/SheerkaComparisonManager.py +++ b/src/core/sheerka/services/SheerkaComparisonManager.py @@ -38,7 +38,7 @@ class SheerkaComparisonManager(BaseService): DEFAULT_COMPARISON_VALUE = 1 def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=14) def initialize(self): cache = ListCache().auto_configure(self.COMPARISON_ENTRY) diff --git a/src/core/sheerka/services/SheerkaConceptManager.py b/src/core/sheerka/services/SheerkaConceptManager.py index 1a39517..32edfee 100644 --- a/src/core/sheerka/services/SheerkaConceptManager.py +++ b/src/core/sheerka/services/SheerkaConceptManager.py @@ -12,8 +12,7 @@ from core.builtin_concepts_ids import BuiltinConcepts, AllBuiltinConcepts, Built from core.builtin_helpers import ensure_concept, ensure_bnf from core.concept import Concept, DEFINITION_TYPE_DEF, DEFINITION_TYPE_BNF, freeze_concept_attrs, ConceptMetadata, \ VARIABLE_PREFIX -from core.error import ErrorObj -from core.global_symbols import EVENT_CONCEPT_CREATED, NotInit, NotFound +from core.global_symbols import EVENT_CONCEPT_CREATED, NotInit, NotFound, ErrorObj, EVENT_CONCEPT_DELETED from core.sheerka.services.sheerka_service import BaseService from core.tokenizer import Tokenizer, TokenKind from sdp.sheerkaDataProvider import SheerkaDataProviderDuplicateKeyError @@ -100,7 +99,7 @@ class SheerkaConceptManager(BaseService): RESOLVED_CONCEPTS_BY_FIRST_KEYWORD_ENTRY = "ConceptManager:Resolved_Concepts_By_First_Keyword" def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=11) self.forbidden_meta = {"is_builtin", "key", "id", "props", "variables"} self.allowed_meta = {attr for attr in vars(ConceptMetadata) if not attr.startswith("_") and attr not in self.forbidden_meta} @@ -147,7 +146,7 @@ class SheerkaConceptManager(BaseService): self.sheerka.om.put(self.sheerka.OBJECTS_IDS_ENTRY, self.USER_CONCEPTS_IDS, 1000) # initialize the dictionary of first tokens - self.sheerka.om.get(self.CONCEPTS_BY_FIRST_KEYWORD_ENTRY, None) # to init the cache with the values from sdp + self.sheerka.om.get(self.CONCEPTS_BY_FIRST_KEYWORD_ENTRY, None) # to init the cache with the values from sdp concepts_by_first_keyword = self.sheerka.om.current_cache_manager().copy(self.CONCEPTS_BY_FIRST_KEYWORD_ENTRY) res = self.resolve_concepts_by_first_keyword(context, concepts_by_first_keyword) self.sheerka.om.put(self.RESOLVED_CONCEPTS_BY_FIRST_KEYWORD_ENTRY, False, res.body) @@ -353,6 +352,8 @@ class SheerkaConceptManager(BaseService): :param concept: :return: """ + # TODO : resolve concept first + sheerka = context.sheerka refs = self.sheerka.om.get(self.CONCEPTS_REFERENCES_ENTRY, concept.id) if refs is not NotFound: @@ -361,6 +362,7 @@ class SheerkaConceptManager(BaseService): try: sheerka.om.remove_concept(concept) + sheerka.publish(context, EVENT_CONCEPT_DELETED, concept) return sheerka.ret(self.NAME, True, sheerka.new(BuiltinConcepts.SUCCESS)) except ConceptNotFound as ex: return sheerka.ret(self.NAME, False, sheerka.err(ex)) diff --git a/src/core/sheerka/services/SheerkaDebugManager.py b/src/core/sheerka/services/SheerkaDebugManager.py index 8fc9d85..081fb6b 100644 --- a/src/core/sheerka/services/SheerkaDebugManager.py +++ b/src/core/sheerka/services/SheerkaDebugManager.py @@ -91,6 +91,14 @@ class BaseDebugLogger: BaseDebugLogger.ids[hint] = 0 return BaseDebugLogger.ids[hint] + @staticmethod + def current_id(hint): + if hint in BaseDebugLogger.ids: + return BaseDebugLogger.ids[hint] + else: + BaseDebugLogger.ids[hint] = 0 + return 0 + def __init__(self, debug_manager, context, who, method_name, debug_id): pass @@ -276,7 +284,7 @@ class SheerkaDebugManager(BaseService): children_activation_regex = re.compile(r"(\d+)\+") def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=1) self.activated = False # is debug activated self.explicit = False # No need to activate context debug when debug mode is on # to remove ? self.context_cache = set() # debug for specific context # to remove ? @@ -295,16 +303,6 @@ class SheerkaDebugManager(BaseService): ] def initialize(self): - # TO REMOVE ??? - self.sheerka.bind_service_method(self.set_explicit, True) - self.sheerka.bind_service_method(self.activate_debug_for, True) - self.sheerka.bind_service_method(self.deactivate_debug_for, True) - self.sheerka.bind_service_method(self.debug_activated, False) - self.sheerka.bind_service_method(self.debug_activated_for, False) - self.sheerka.bind_service_method(self.get_context_debug_mode, False) - self.sheerka.bind_service_method(self.debug_rule_activated, False) - self.sheerka.bind_service_method(self.debug, False, visible=False) - self.sheerka.bind_service_method(self.set_debug, True) self.sheerka.bind_service_method(self.inspect, False) self.sheerka.bind_service_method(self.get_debugger, False) @@ -341,79 +339,13 @@ class SheerkaDebugManager(BaseService): self.sheerka.record_var(context, self.NAME, "activated", self.activated) return self.sheerka.ret(SheerkaDebugManager.NAME, True, self.sheerka.new(BuiltinConcepts.SUCCESS)) - def set_explicit(self, context, value=True): - self.explicit = value - self.sheerka.record_var(context, self.NAME, "explicit", self.explicit) - return self.sheerka.ret(SheerkaDebugManager.NAME, True, self.sheerka.new(BuiltinConcepts.SUCCESS)) - - def activate_debug_for(self, context, debug_id, children=False): - """ - - :param context: - :param debug_id: if debug_id is str, activate variable cache, context_cache otherwise - :param children: - :return: - """ - # preprocess - if isinstance(debug_id, str) and (m := self.children_activation_regex.match(debug_id)): - debug_id = int(m.group(1)) - children = True - - if isinstance(debug_id, str): - self.variable_cache.add(debug_id) - self.sheerka.record_var(context, self.NAME, "variable_cache", self.variable_cache) - else: - self.context_cache.add(debug_id) - if children: - self.context_cache.add(str(debug_id) + "+") - self.sheerka.record_var(context, self.NAME, "context_cache", self.context_cache) - return self.sheerka.ret(SheerkaDebugManager.NAME, True, self.sheerka.new(BuiltinConcepts.SUCCESS)) - - def deactivate_debug_for(self, context, debug_id, children=False): - if isinstance(debug_id, str): - self.variable_cache.discard(debug_id) - self.sheerka.record_var(context, self.NAME, "variable_cache", self.variable_cache) - else: - self.context_cache.discard(debug_id) - if children: - self.context_cache.discard(str(debug_id) + "+") - self.sheerka.record_var(context, self.NAME, "context_cache", self.context_cache) - return self.sheerka.ret(SheerkaDebugManager.NAME, True, self.sheerka.new(BuiltinConcepts.SUCCESS)) - - def debug_activated(self): - return self.activated - - def debug_activated_for(self, debug_id): - if not self.activated: - return None - - return debug_id in self.variable_cache - - def debug_rule_activated(self, rule_id, context_id): - """ - - :param rule_id: - :param context_id: - :return: - """ - key = f"{rule_id}|{context_id}" - return key in self.rules_cache - - def get_context_debug_mode(self, context_id): - if not self.activated: - return None, None - - debug_for_children = "protected" if str(context_id) + "+" in self.context_cache else None - debug_for_self = "private" if not self.explicit or context_id in self.context_cache else None - - return debug_for_self, debug_for_children - def debug(self, *args, **kwargs): print(*args, **kwargs) - def get_debugger(self, context, who, method_name): + def get_debugger(self, context, who, method_name, new_debug_id=True): if self.compute_debug(context, who, method_name): - debug_id = ConsoleDebugLogger.next_id(context.event.get_digest() + str(context.id)) + debug_id = ConsoleDebugLogger.next_id(context.event.get_digest() + str(context.id)) if new_debug_id \ + else ConsoleDebugLogger.current_id(context.event.get_digest() + str(context.id)) return ConsoleDebugLogger(self, context, who, method_name, debug_id) return NullDebugLogger() diff --git a/src/core/sheerka/services/SheerkaEvaluateConcept.py b/src/core/sheerka/services/SheerkaEvaluateConcept.py index b066cf5..1894142 100644 --- a/src/core/sheerka/services/SheerkaEvaluateConcept.py +++ b/src/core/sheerka/services/SheerkaEvaluateConcept.py @@ -11,7 +11,8 @@ from core.sheerka.services.sheerka_service import BaseService from core.tokenizer import Tokenizer from core.utils import unstr_concept from parsers.BaseNodeParser import ConceptNode -from parsers.ExpressionParser import ExpressionParser, TrueifyVisitor +from parsers.ExpressionParser import ExpressionParser +from parsers.expressions import TrueifyVisitor CONCEPT_EVALUATION_STEPS = [ BuiltinConcepts.BEFORE_EVALUATION, diff --git a/src/core/sheerka/services/SheerkaEvaluateRules.py b/src/core/sheerka/services/SheerkaEvaluateRules.py index d1ec1cb..520dea3 100644 --- a/src/core/sheerka/services/SheerkaEvaluateRules.py +++ b/src/core/sheerka/services/SheerkaEvaluateRules.py @@ -1,7 +1,9 @@ from core.builtin_concepts import BuiltinConcepts from core.builtin_helpers import expect_one +from core.global_symbols import EVENT_RULE_CREATED, EVENT_RULE_DELETED, EVENT_RULE_ID_DELETED from core.sheerka.services.sheerka_service import BaseService from evaluators.ConceptEvaluator import ConceptEvaluator +from sheerkarete.network import ReteNetwork DISABLED_RULES = "#disabled#" LOW_PRIORITY_RULES = "#low_priority#" @@ -11,12 +13,18 @@ class SheerkaEvaluateRules(BaseService): NAME = "EvaluateRules" def __init__(self, sheerka): - super().__init__(sheerka) + # order must be before RuleManager because of event subscription + super().__init__(sheerka, 4) self.evaluators_by_name = None + self.network = ReteNetwork() def initialize(self): - self.sheerka.bind_service_method(self.evaluate_format_rules, False) + self.sheerka.bind_service_method(self.evaluate_format_rules, False, visible=False) + self.sheerka.bind_service_method(self.evaluate_exec_rules, False, visible=False) self.reset_evaluators() + self.sheerka.subscribe(EVENT_RULE_CREATED, self.on_rule_created) + self.sheerka.subscribe(EVENT_RULE_DELETED, self.on_rule_deleted) + self.sheerka.subscribe(EVENT_RULE_ID_DELETED, self.on_rule_deleted) def reset_evaluators(self): # instantiate evaluators, once for all, only keep when it's enabled @@ -24,6 +32,20 @@ class SheerkaEvaluateRules(BaseService): evaluators = [e for e in evaluators if e.enabled] self.evaluators_by_name = {e.short_name: e for e in evaluators} + def evaluate_exec_rules(self, context, return_values): + # self.network.add_obj("__rets", return_values) + for ret in return_values: + self.network.add_obj("__ret", ret) + + results = [] # list of return values, for activated rules + for match in self.network.matches: + for rule in match.pnode.rules: + body = context.sheerka.new(BuiltinConcepts.RULE_EVALUATION_RESULT, rule=rule) + return_value = context.sheerka.ret(self.NAME, True, body) + results.append(return_value) + + return results + def evaluate_format_rules(self, context, bag, disabled): return self.evaluate_rules(context, self.sheerka.get_format_rules(), bag, disabled) @@ -37,7 +59,7 @@ class SheerkaEvaluateRules(BaseService): :param disabled: disabled rules (because they have already been fired or whatever) :return: { True : list of success, False :list of failed, '#disabled"': list of disabled...} """ - with context.push(BuiltinConcepts.EVALUATING_RULES, bag, desc="Evaluating rules...") as sub_context: + with context.push(BuiltinConcepts.RULES_EVALUATION, bag, desc="Evaluating rules...") as sub_context: sub_context.protected_hints.add(BuiltinConcepts.EVAL_BODY_REQUESTED) sub_context.protected_hints.add(BuiltinConcepts.EVAL_WHERE_REQUESTED) sub_context.protected_hints.add(BuiltinConcepts.EVAL_UNTIL_SUCCESS_REQUESTED) @@ -81,7 +103,7 @@ class SheerkaEvaluateRules(BaseService): """ results = [] - for rule_predicate in rule.compiled_predicate: + for rule_predicate in rule.compiled_predicates: if rule_predicate.source in bag: # simple case where the rule is an item of the bag. No need of complicate evaluation @@ -94,15 +116,44 @@ class SheerkaEvaluateRules(BaseService): rule_predicate.concept.get_metadata().is_evaluated = False evaluator = self.evaluators_by_name[rule_predicate.evaluator] - results.append(evaluator.eval(context, rule_predicate.predicate)) + res = evaluator.eval(context, rule_predicate.predicate) + if res.status and isinstance(res.body, bool) and res.body: + # one successful value found. No need to look any further + results = [res] + break + else: + results.append(res) - debugger = context.get_debugger(SheerkaEvaluateRules.NAME, "evaluate_rule") + debugger = context.get_debugger(SheerkaEvaluateRules.NAME, "evaluate_rule", new_debug_id=False) debugger.debug_rule(rule, results) - # if context.sheerka.debug_rule_activated(rule_id, context.id): - # context.debug(SheerkaEvaluateRules.NAME, "evaluate_rules", f"result(#{rule_id})", results) return expect_one(context, results) + def remove_from_rete_memory(self, lst): + if lst is None: + return + + for obj in lst: + self.network.remove_obj(obj) + + def on_rule_created(self, context, rule): + """ + When a new rule is added to the system, update the network + """ + if rule.metadata.is_enabled and rule.rete_disjunctions: + self.network.add_rule(rule) + + def on_rule_deleted(self, context, rule): + """ + When a rule is deleted from the system, remove it from the network + """ + if isinstance(rule, str): + rule = self.sheerka.get_rule_by_id(rule) + if not self.sheerka.is_known(rule): + return + + self.network.remove_rule(rule) + @staticmethod def get_debug_format(result): """ diff --git a/src/core/sheerka/services/SheerkaEventManager.py b/src/core/sheerka/services/SheerkaEventManager.py index 998d757..57c1c4e 100644 --- a/src/core/sheerka/services/SheerkaEventManager.py +++ b/src/core/sheerka/services/SheerkaEventManager.py @@ -11,7 +11,7 @@ class SheerkaEventManager(BaseService): NAME = "EventManager" def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=2) self._lock = RLock() self.subscribers = {} diff --git a/src/core/sheerka/services/SheerkaExecute.py b/src/core/sheerka/services/SheerkaExecute.py index 81abdd1..7212eb5 100644 --- a/src/core/sheerka/services/SheerkaExecute.py +++ b/src/core/sheerka/services/SheerkaExecute.py @@ -1,5 +1,4 @@ import core.utils -from cache.Cache import Cache from cache.FastCache import FastCache from core.builtin_concepts import BuiltinConcepts, ReturnValueConcept from core.global_symbols import NotFound @@ -16,6 +15,8 @@ EVALUATOR_STEPS = [ BuiltinConcepts.BEFORE_RENDERING, BuiltinConcepts.RENDERING, BuiltinConcepts.AFTER_RENDERING, + BuiltinConcepts.BEFORE_RULES_EVALUATION, + BuiltinConcepts.AFTER_RULES_EVALUATION, ] @@ -120,6 +121,10 @@ class ParserInput: return self.pos < self.end def the_token_after(self, skip_whitespace=True): + """ + Returns the token after the current one + Never returns None (returns TokenKind.EOF instead) + """ my_pos = self.pos + 1 if my_pos >= self.end: return Token(TokenKind.EOF, "", -1, -1, -1) @@ -167,7 +172,8 @@ class SheerkaExecute(BaseService): PARSERS_INPUTS_ENTRY = "Execute:ParserInput" # entry for admin or internal variables def __init__(self, sheerka): - super().__init__(sheerka) + # order must be after SheerkaEvaluateRules because of self.rules_evaluation_service + super().__init__(sheerka, order=5) self.pi_cache = FastCache(default=lambda key: ParserInput(key), max_size=20) self.instantiated_evaluators = None self.evaluators_by_name = None @@ -191,12 +197,18 @@ class SheerkaExecute(BaseService): # Except 2 : we store the type of the parser, not its instance self.grouped_parsers_cache = {} + self.rules_eval_service = None + def initialize(self): - self.sheerka.bind_service_method(self.execute, True) + self.sheerka.bind_service_method(self.execute, True, visible=False) + self.sheerka.bind_service_method(self.execute_rules, True, visible=False) self.reset_registered_evaluators() self.reset_registered_parsers() + from core.sheerka.services.SheerkaEvaluateRules import SheerkaEvaluateRules + self.rules_eval_service = self.sheerka.services[SheerkaEvaluateRules.NAME] + def reset_state(self): self.pi_cache.clear() @@ -347,7 +359,7 @@ class SheerkaExecute(BaseService): if pi is NotFound: # when CacheManager.cache_only is True pi = ParserInput(text) self.pi_cache.put(text, pi) - return pi + return ParserInput(text, pi.tokens) # new instance, but no need to tokenize the text again key = text or core.utils.get_text_from_tokens(tokens) pi = ParserInput(key, tokens) @@ -582,6 +594,55 @@ class SheerkaExecute(BaseService): return return_values + def execute_rules(self, context, return_values, rules_steps, evaluation_steps): + """" + Executes the execution rules until no match is found + :param context: + :param return_values: input return values + :param rules_steps: steps are configurable + :param evaluation_steps: steps are configurable + :return: out return_values + """ + continue_execution = True + counter = 0 + in_rete_memory = None + while continue_execution: + with context.push(BuiltinConcepts.PROCESSING, {"counter": counter}, desc=f"{counter=}") as sub_context: + + # apply rule evaluation steps + for step in rules_steps: + if step == BuiltinConcepts.RULES_EVALUATION: + eval_res = self.rules_eval_service.evaluate_exec_rules(sub_context, return_values) + if not eval_res: + self.rules_eval_service.remove_from_rete_memory(return_values) + continue_execution = False + break + else: + in_rete_memory = return_values.copy() + return_values = eval_res + else: + return_values = self.call_evaluators(sub_context, return_values, step) + + if not continue_execution: + break + + # evaluate the result + return_values = [r.body.body.compiled_action for r in return_values] + while True: + copy = return_values[:] + for step in evaluation_steps: + return_values = self.call_evaluators(sub_context, return_values, step) + + if copy == return_values[:]: + break + + # evaluation is done. Remove object in Rete memory + self.rules_eval_service.remove_from_rete_memory(in_rete_memory) + + counter += 1 + + return return_values + def undo_preprocess(self): for item, var_name, value in self.old_values: setattr(item, var_name, value) diff --git a/src/core/sheerka/services/SheerkaHasAManager.py b/src/core/sheerka/services/SheerkaHasAManager.py index b0b5726..5da6b8c 100644 --- a/src/core/sheerka/services/SheerkaHasAManager.py +++ b/src/core/sheerka/services/SheerkaHasAManager.py @@ -7,7 +7,7 @@ class SheerkaHasAManager(BaseService): NAME = "HasAManager" def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=22) def initialize(self): self.sheerka.bind_service_method(self.set_hasa, True) diff --git a/src/core/sheerka/services/SheerkaIsAManager.py b/src/core/sheerka/services/SheerkaIsAManager.py index ac5b073..8682c92 100644 --- a/src/core/sheerka/services/SheerkaIsAManager.py +++ b/src/core/sheerka/services/SheerkaIsAManager.py @@ -15,7 +15,7 @@ class SheerkaIsAManager(BaseService): CONCEPTS_IN_GROUPS_ENTRY = "IsAManager:Concepts_In_Groups" # cache for get_set_elements() def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=21) def initialize(self): self.sheerka.bind_service_method(self.set_isa, True) diff --git a/src/core/sheerka/services/SheerkaMemory.py b/src/core/sheerka/services/SheerkaMemory.py index 2e5da33..6deb466 100644 --- a/src/core/sheerka/services/SheerkaMemory.py +++ b/src/core/sheerka/services/SheerkaMemory.py @@ -20,7 +20,7 @@ class SheerkaMemory(BaseService): OBJECTS_ENTRY = "Memory:Objects" def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=13) self.short_term_objects = FastCache() self.registration = {} @@ -39,6 +39,8 @@ class SheerkaMemory(BaseService): cache = ListIfNeededCache().auto_configure(self.OBJECTS_ENTRY) self.sheerka.om.register_cache(self.OBJECTS_ENTRY, cache, persist=True, use_ref=True) + + self.sheerka.subscribe(EVENT_CONTEXT_DISPOSED, self.remove_context) def reset(self): self.short_term_objects.clear() @@ -48,9 +50,6 @@ class SheerkaMemory(BaseService): self.short_term_objects.clear() self.registration.clear() - def initialize_deferred(self, context, is_first_time): - self.sheerka.subscribe(EVENT_CONTEXT_DISPOSED, self.remove_context) - def get_from_short_term_memory(self, context, key): while True: try: diff --git a/src/core/sheerka/services/SheerkaOut.py b/src/core/sheerka/services/SheerkaOut.py index d646406..39f6e70 100644 --- a/src/core/sheerka/services/SheerkaOut.py +++ b/src/core/sheerka/services/SheerkaOut.py @@ -31,6 +31,7 @@ class SheerkaOut(BaseService): if valid_rules: if len(valid_rules) > 1: # TODO manage when too many rules + print("TODO: TOO MANY RULES !!!!!") pass rule = valid_rules[0] diff --git a/src/core/sheerka/services/SheerkaResultManager.py b/src/core/sheerka/services/SheerkaResultManager.py index f877853..0123dd2 100644 --- a/src/core/sheerka/services/SheerkaResultManager.py +++ b/src/core/sheerka/services/SheerkaResultManager.py @@ -37,11 +37,12 @@ class SheerkaResultManager(BaseService): self.sheerka.bind_service_method(self.get_last_created_concept, False, as_name="last_created_concept") self.sheerka.bind_service_method(self.get_last_error, False, as_name="last_err") - def initialize_deferred(self, context, is_first_time): - self.restore_values(*self.state_vars) self.sheerka.subscribe(EVENT_USER_INPUT_EVALUATED, self.user_input_evaluated) self.sheerka.subscribe(EVENT_CONCEPT_CREATED, self.new_concept_created) + def initialize_deferred(self, context, is_first_time): + self.restore_values(*self.state_vars) + def test_only_reset(self): self.executions_contexts_cache.clear() self.last_execution = None diff --git a/src/core/sheerka/services/SheerkaRuleManager.py b/src/core/sheerka/services/SheerkaRuleManager.py index a83021e..496a57e 100644 --- a/src/core/sheerka/services/SheerkaRuleManager.py +++ b/src/core/sheerka/services/SheerkaRuleManager.py @@ -1,28 +1,35 @@ import operator import re from dataclasses import dataclass -from typing import Union +from typing import Union, Set, List from cache.Cache import Cache from cache.ListIfNeededCache import ListIfNeededCache from core.builtin_concepts import BuiltinConcepts, ReturnValueConcept -from core.builtin_helpers import parse_unrecognized, only_successful, ensure_rule +from core.builtin_helpers import parse_unrecognized, is_a_question, parse_python, \ + ensure_evaluated, expect_one, parse_expression from core.concept import Concept -from core.global_symbols import EVENT_RULE_PRECEDENCE_MODIFIED, RULE_COMPARISON_CONTEXT, NotFound -from core.rule import Rule -from core.sheerka.services.sheerka_service import BaseService +from core.global_symbols import EVENT_RULE_PRECEDENCE_MODIFIED, RULE_COMPARISON_CONTEXT, NotFound, ErrorObj, \ + EVENT_RULE_CREATED, EVENT_RULE_DELETED +from core.rule import Rule, ACTION_TYPE_PRINT +from core.sheerka.Sheerka import RECOGNIZED_BY_NAME, RECOGNIZED_BY_ID +from core.sheerka.services.sheerka_service import BaseService, FailedToCompileError from core.tokenizer import Keywords, TokenKind, Token, IterParser from core.utils import index_tokens, COLORS, get_text_from_tokens from evaluators.ConceptEvaluator import ConceptEvaluator -from evaluators.PythonEvaluator import PythonEvaluator +from evaluators.PythonEvaluator import PythonEvaluator, Expando from parsers.BaseNodeParser import SourceCodeWithConceptNode, ConceptNode, SourceCodeNode +from parsers.ExpressionParser import AndNode, ExpressionParser from parsers.PythonParser import PythonNode +from sheerkarete.conditions import AndConditions + +CONCEPTS_ONLY_PARSERS = ["ExactConcept", "Bnf", "Sya", "Sequence"] identifier_regex = re.compile(r"[\w _.]+") @dataclass -class FormatRuleError: +class FormatRuleError(ErrorObj): pass @@ -199,7 +206,7 @@ class FormatAstMulti(FormatAstNode): **kwargs) -class FormatRuleParser(IterParser): +class FormatRuleActionParser(IterParser): @staticmethod def to_text(list_or_dict_of_tokens): @@ -242,7 +249,7 @@ class FormatRuleParser(IterParser): def parse(self): """ - Parses a format rule + Parses the print part of the format rule format ::= {variable'} | function(...) | rawtext :return: """ @@ -394,7 +401,7 @@ class FormatRuleParser(IterParser): source = get_text_from_tokens(args[0]) if len(source) > 1 and source[0] in ("'", '"') and source[-1] in ("'", '"'): source = source[1:-1] - parser = FormatRuleParser(source) + parser = FormatRuleActionParser(source) res = parser.parse() self.error_sink = parser.error_sink return FormatAstColor(color, res) @@ -497,12 +504,139 @@ class FormatRuleParser(IterParser): return FormatAstMulti(get_text_from_tokens(args[0])) +@dataclass +class EmitPythonCodeException(Exception): + error: object + + +class PythonCodeEmitter: + + def __init__(self, context, text=None): + self.context = context + self.text = text or "" + self.var_counter = 0 + self.variables = [] + + def add(self, text): + self.text += f" and {text}" if self.text else text + return self + + def recognize(self, obj, as_name, root=True): + if isinstance(obj, str): + return self.recognize_str(obj, as_name) + elif isinstance(obj, (int, float)): + return self.recognize_int(obj, as_name) + elif isinstance(obj, Concept): + return self.recognize_concept(obj, as_name, root) + elif isinstance(obj, Expando): + return self.recognize_expando(obj, as_name, root) + else: + raise NotImplementedError() + + def recognize_str(self, text, as_name): + if self.text: + self.text += " and " + + if "'" in text and '"' in text: + self.text += f"{as_name} == '{text}'" + elif "'" in text: + self.text += f'{as_name} == "{text}"' + else: + self.text += f"{as_name} == '{text}'" + return self + + def recognize_int(self, value, as_name): + if self.text: + self.text += " and " + + self.text += f"{as_name} == {value}" + return self + + def recognize_expando(self, value, as_name, root=True): + if self.text: + self.text += " and " + + if not root: + as_name = self.add_variable(as_name) + + self.text += f"isinstance({as_name}, Expando) and {as_name}.get_name() == '{value.get_name()}'" + return self + + def recognize_concept(self, concept, as_name, root=True): + if self.text: + self.text += " and " + + if not root: + as_name = self.add_variable(as_name) + + if concept.get_hint(BuiltinConcepts.RECOGNIZED_BY) == RECOGNIZED_BY_NAME: + self.text += f"isinstance({as_name}, Concept) and {as_name}.name == '{concept.name}'" + elif concept.get_hint(BuiltinConcepts.RECOGNIZED_BY) == RECOGNIZED_BY_ID: + self.text += f"isinstance({as_name}, Concept) and {as_name}.id == '{concept.id}'" + else: + self.text += f"isinstance({as_name}, Concept) and {as_name}.key == '{concept.key}'" + if len(concept.get_metadata().variables) > 0: + # add variables constraints + evaluated = ensure_evaluated(self.context, concept, eval_body=False, metadata=["variables"]) + + if not self.context.sheerka.is_success(evaluated) and evaluated.key != concept.key: + raise EmitPythonCodeException(evaluated) + + for k, v in concept.variables().items(): + self.recognize(v, f"{as_name}.get_value('{k}')", root=False) + + return self + + def add_variable(self, target): + var_name = f"__x_{self.var_counter:02}__" + self.var_counter += 1 + self.variables.append((var_name, target)) + return var_name + + def get_text(self): + if self.variables: + variables_as_str = '\n'.join([f"{k} = {v}" for k, v in self.variables]) + return variables_as_str + "\n" + self.text + + return self.text + + +class NoConditionFound(ErrorObj): + def __eq__(self, other): + return isinstance(other, NoConditionFound) + + def __hash__(self): + return 0 + + @dataclass() -class RulePredicate: - source: str - evaluator: str - predicate: ReturnValueConcept - concept: Union[Concept, None] +class RuleCompiledPredicate: + """ + The 'when' expression is parsed to have a ReturnValueConcept or a Concept that can then be evaluated + Depending on the evaluator, the 'predicate' attribute or the 'concept' attribute will be used + """ + source: str # what was compiled # DO NOT REMOVE + action: str # sheerka action when the rule must be executed # can be removed + + # when used as a list of predicate to iterate thru + evaluator: str # evaluator to use when the rule will be evaluated + predicate: ReturnValueConcept # compiled source as ReturnValue + concept: Union[Concept, None] # compiled source as concept + + variables: Set[str] = None # TODO: set of required variables + + +@dataclass +class CompiledWhenResult: + """ + For a given source to compile (a given 'when') + List of RuleCompiledPredicate found + and list of Rete Conditions + + The two ways of evaluating a 'when' are used by Sheerka + """ + compiled_predicates: List[RuleCompiledPredicate] + rete_disjunctions: List[AndConditions] class SheerkaRuleManager(BaseService): @@ -513,15 +647,18 @@ class SheerkaRuleManager(BaseService): RULES_BY_NAME_ENTRY = "RuleManager:Rules_By_Name" def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=12) self._format_rules = None # sorted by priority + self._exec_rules = None # sorted by priority + self.expression_parser = ExpressionParser() def initialize(self): self.sheerka.bind_service_method(self.create_new_rule, True, visible=False) + self.sheerka.bind_service_method(self.remove_rule, True) self.sheerka.bind_service_method(self.get_rule_by_id, False) self.sheerka.bind_service_method(self.get_rule_by_name, False) - self.sheerka.bind_service_method(self.dump_desc_rule, False, as_name="desc_rule") self.sheerka.bind_service_method(self.get_format_rules, False, visible=False) + self.sheerka.bind_service_method(self.get_exec_rules, False, visible=False) self.sheerka.bind_service_method(self.resolve_rule, False, visible=False) cache = Cache().auto_configure(self.FORMAT_RULE_ENTRY) @@ -531,6 +668,8 @@ class SheerkaRuleManager(BaseService): cache = ListIfNeededCache().auto_configure(self.RULES_BY_NAME_ENTRY) self.sheerka.om.register_cache(self.RULES_BY_NAME_ENTRY, cache, True, True) + self.sheerka.subscribe(EVENT_RULE_PRECEDENCE_MODIFIED, self.update_rules_priorities) + def initialize_deferred(self, context, is_first_time): if is_first_time: @@ -551,15 +690,17 @@ class SheerkaRuleManager(BaseService): # compile all format the rules for rule_id, rule_def in self.sheerka.om.get_all(self.FORMAT_RULE_ENTRY, cache_only=True).items(): - rule = self.init_rule(context, rule_def) + self.init_rule(context, rule_def) + + for rule_id, rule_def in self.sheerka.om.get_all(self.EXEC_RULE_ENTRY, cache_only=True).items(): + self.init_rule(context, rule_def) # update rules priorities self.update_rules_priorities(context) - self.sheerka.subscribe(EVENT_RULE_PRECEDENCE_MODIFIED, self.update_rules_priorities) - def reset_state(self): self._format_rules = None + self._exec_rules = None def update_rules_priorities(self, context): """ @@ -575,50 +716,93 @@ class SheerkaRuleManager(BaseService): rule.priority = rules_weights[rule.str_id] self._format_rules = None + self._exec_rules = None def init_rule(self, context, rule: Rule): if rule.metadata.is_compiled: - return + return rule - if rule.compiled_predicate is None: - res = self.compile_when(context, self.NAME, rule.metadata.predicate) - if not isinstance(res, list): - rule.error_sink = [res.body] - return - rule.compiled_predicate = res + if rule.compiled_predicates is None: + try: + compiled_result = self.compile_when(context, self.NAME, rule.metadata.predicate) + rule.compiled_predicates = compiled_result.compiled_predicates + rule.rete_disjunctions = compiled_result.rete_disjunctions + except FailedToCompileError as ex: + rule.compiled_predicates = None + rule.rete_disjunctions = None + rule.error_sink = {"when": ex.cause} if rule.compiled_action is None: - res = self.compile_print(context, rule.metadata.action) - if not res.status: - rule.error_sink = [res.body] - return - rule.compiled_action = res.body + compile_method = self.compile_print if rule.metadata.action_type == ACTION_TYPE_PRINT else self.compile_exec - # rule.variables = self.get_variables() + res = compile_method(context, rule.metadata.action) + if not res.status: + rule.compiled_action = None + if rule.error_sink is None: + rule.error_sink = {rule.metadata.action_type: res.body} + else: + rule.error_sink[rule.metadata.action_type] = res.body + else: + rule.compiled_action = res.body if rule.metadata.action_type == ACTION_TYPE_PRINT else res rule.metadata.is_compiled = True - rule.metadata.is_enabled = True + rule.metadata.is_enabled = rule.error_sink is None + return rule - def compile_when(self, context, name, source): - # parser_input = self.sheerka.services[SheerkaExecute.NAME].get_parser_input(source) - parsed = parse_unrecognized(context, - source, - parsers="all", - who=name, - prop=Keywords.WHEN, - filter_func=only_successful) + def compile_when(self, context, who, source): + """ + Compile the predicate + :param context: + :param who: service which requested the compilation + :param source: what to compile + """ - if not parsed.status: - return parsed + # first, try to parse using expression parser + # -> Detect xxx and yyy or not zzz - if self.sheerka.isinstance(parsed.body, BuiltinConcepts.ONLY_SUCCESSFUL): - parsed = parsed.body.body + action = None + parsed = [] + errors = [] + all_rete_disjunctions = [] + parsed_expr_ret = parse_expression(context, source) + if parsed_expr_ret.status: + conjunctions = parsed_expr_ret.body.body.parts if isinstance(parsed_expr_ret.body.body, AndNode) else \ + [parsed_expr_ret.body.body] - return self.add_evaluators(source, parsed if hasattr(parsed, "__iter__") else [parsed]) + # recognize __action == '' + if (action := self._recognized_action_definition(conjunctions[0].tokens)) is not None: + conjunctions = conjunctions[1:] + + if len(conjunctions) == 0: + errors.append(NoConditionFound()) + else: + # compile conditions + try: + return_values, rete_disjunctions = self.expression_parser.compile_conjunctions(context, + conjunctions, + who) + + parsed.extend(return_values) + all_rete_disjunctions.extend(rete_disjunctions) + + except FailedToCompileError as ex: + errors.append(ex.cause) + + if len(parsed) == 0: + raise FailedToCompileError(errors) + + try: + compiled_predicates = self.add_evaluators(context, + source, + action, + parsed if hasattr(parsed, "__iter__") else [parsed]) + return CompiledWhenResult(compiled_predicates, all_rete_disjunctions) + except EmitPythonCodeException as ex: + raise FailedToCompileError([ex.error]) def compile_print(self, context, source): - parser = FormatRuleParser(source) + parser = FormatRuleActionParser(source) parsed = parser.parse() if parser.error_sink: return self.sheerka.ret(self.NAME, @@ -627,6 +811,16 @@ class SheerkaRuleManager(BaseService): else: return self.sheerka.ret(self.NAME, True, parsed) + def compile_exec(self, context, source): + parsed = parse_unrecognized(context, + source, + parsers="all", + who=self.NAME, + prop=Keywords.THEN, + filter_func=expect_one) + + return parsed + def set_id_if_needed(self, rule: Rule): """ Set the id for the concept if needed @@ -649,25 +843,52 @@ class SheerkaRuleManager(BaseService): # set id before saving in db self.set_id_if_needed(rule) - if rule.compiled_predicate and rule.compiled_action: + if rule.compiled_predicates and rule.compiled_action: rule.metadata.is_compiled = True rule.metadata.is_enabled = True # save it - if rule.metadata.action_type == "print": - self.sheerka.om.put(self.FORMAT_RULE_ENTRY, rule.metadata.id, rule) + if rule.metadata.action_type == ACTION_TYPE_PRINT: + sheerka.om.put(self.FORMAT_RULE_ENTRY, rule.metadata.id, rule) self._format_rules = None else: - self.sheerka.om.put(self.EXEC_RULE_ENTRY, rule.metadata.id, rule) + sheerka.om.put(self.EXEC_RULE_ENTRY, rule.metadata.id, rule) + self._exec_rules = None # save by name if needed if rule.metadata.name: self.sheerka.om.put(self.RULES_BY_NAME_ENTRY, rule.metadata.name, rule) + # rule is created. publish the event + sheerka.publish(context, EVENT_RULE_CREATED, rule) + # process the return if needed ret = sheerka.ret(self.NAME, True, sheerka.new(BuiltinConcepts.NEW_RULE, body=rule)) return ret + def remove_rule(self, context, rule): + """ + Remove a rule + """ + rule = self.resolve_rule(context, rule) + if rule is None: + return + + # rule will be deleted. publish the event first, as the rule may not be available after + self.sheerka.publish(context, EVENT_RULE_DELETED, rule) + + if rule.metadata.action_type == ACTION_TYPE_PRINT: + self.sheerka.om.delete(self.FORMAT_RULE_ENTRY, rule.metadata.id) + self._format_rules = None + else: + self.sheerka.om.delete(self.EXEC_RULE_ENTRY, rule.metadata.id) + self._exec_rules = None + + if rule.metadata.name: + self.sheerka.om.delete(self.RULES_BY_NAME_ENTRY, rule.metadata.name) + + return self.sheerka.ret(self.NAME, True, self.sheerka.new(BuiltinConcepts.SUCCESS)) + def init_builtin_rules(self, context): # self.sheerka.init_log.debug("Initializing default rules") rules = [ @@ -760,29 +981,6 @@ class SheerkaRuleManager(BaseService): return rule - def dump_desc_rule(self, rules): - """ - dumps the definition of a rule - :param rules: - :return: - """ - ensure_rule(rules) - - if not hasattr(rules, "__iter__"): - rules = [rules] - - first = True - for rule in rules: - if not first: - self.sheerka.log.info("") - self.sheerka.log.info(f"id : {rule.id}") - self.sheerka.log.info(f"name : {rule.metadata.name}") - self.sheerka.log.info(f"type : {rule.metadata.action_type}") - self.sheerka.log.info(f"predicate : {rule.metadata.predicate}") - self.sheerka.log.info(f"action : {rule.metadata.action}") - self.sheerka.log.info(f"compiled : {rule.metadata.is_compiled}") - self.sheerka.log.info(f"enabled : {rule.metadata.is_enabled}") - def get_format_rules(self): if self._format_rules: return self._format_rules @@ -792,32 +990,56 @@ class SheerkaRuleManager(BaseService): reverse=True) return self._format_rules - def add_evaluators(self, source, ret_vals): + def get_exec_rules(self): + if self._exec_rules: + return self._exec_rules + + self._exec_rules = sorted(self.sheerka.om.list(self.EXEC_RULE_ENTRY, cache_only=True), + key=operator.attrgetter('priority'), + reverse=True) + return self._exec_rules + + def add_evaluators(self, context, source, action, ret_vals): """ Browse the ReturnValueConcepts to determine the evaluator to use - Returns a list of tuple (evaluator_name, return_value) + Returns a list of RulePredicate, basically a tuple (evaluator_name, return_value) + :param context: :param source: :param ret_vals: :return: """ + + def get_rule_predicate_from_concept(c): + if is_a_question(context, c): + return RuleCompiledPredicate(source, action, ConceptEvaluator.NAME, r, c) + else: + to_parse = PythonCodeEmitter(context, "__ret.status").recognize_concept(c, "__ret.body").get_text() + return RuleCompiledPredicate(source, action, PythonEvaluator.NAME, parse_python(context, to_parse), + None) + res = [] for r in ret_vals: underlying = self.sheerka.objvalue(r) if isinstance(underlying, PythonNode): - res.append(RulePredicate(source, PythonEvaluator.NAME, r, None)) + res.append(RuleCompiledPredicate(source, action, PythonEvaluator.NAME, r, None)) elif isinstance(underlying, SourceCodeWithConceptNode): - res.append(RulePredicate(source, PythonEvaluator.NAME, r, None)) + res.append(RuleCompiledPredicate(source, action, PythonEvaluator.NAME, r, None)) elif isinstance(underlying, SourceCodeNode): - res.append(RulePredicate(source, PythonEvaluator.NAME, r, None)) + res.append(RuleCompiledPredicate(source, action, PythonEvaluator.NAME, r, None)) elif isinstance(underlying, Concept): - res.append(RulePredicate(source, ConceptEvaluator.NAME, r, underlying)) + res.append(get_rule_predicate_from_concept(underlying)) elif hasattr(underlying, "__iter__") and len(underlying) == 1 and isinstance(underlying[0], ConceptNode): - res.append(RulePredicate(source, ConceptEvaluator.NAME, r, underlying[0].concept)) + res.append(get_rule_predicate_from_concept(underlying[0].concept)) else: raise NotImplementedError(r) return res def resolve_rule(self, context, obj): + """ + Given obj, try to find the corresponding rule + :param context: + :param obj: + """ if obj is None: return None @@ -841,7 +1063,7 @@ class SheerkaRuleManager(BaseService): (rule := self._inner_get_by_id(str(rule_id))) is not None: return rule else: - return obj + return self._inner_get_by_id(obj.id) return None @@ -855,3 +1077,41 @@ class SheerkaRuleManager(BaseService): return rule return None + + @staticmethod + def _recognized_action_definition(tokens): + """ + Tries to recognize the pattern __action = xxx in the tokens + """ + iter_token = iter(tokens) + try: + token = next(iter_token) + if token.value != "__action": + return None + token = next(iter_token) + if token.type == TokenKind.WHITESPACE: + token = next(iter_token) + if token.type != TokenKind.EQUALSEQUALS: + return None + token = next(iter_token) + if token.type == TokenKind.WHITESPACE: + token = next(iter_token) + if token.type == TokenKind.STRING: + return token.strip_quote + except StopIteration: + pass + + return None + + @staticmethod + def _get_parsed_concept(context, return_value): + if not context.sheerka.isinstance(return_value.body, BuiltinConcepts.PARSER_RESULT): + return None + + if isinstance(return_value.body.body, Concept): + return return_value.body.body + + if isinstance(return_value.body.body, ConceptNode): + return return_value.body.body.concept + + return None diff --git a/src/core/sheerka/services/SheerkaVariableManager.py b/src/core/sheerka/services/SheerkaVariableManager.py index 32cb3b8..e1ca65b 100644 --- a/src/core/sheerka/services/SheerkaVariableManager.py +++ b/src/core/sheerka/services/SheerkaVariableManager.py @@ -41,9 +41,9 @@ class SheerkaVariableManager(BaseService): INTERNAL_VARIABLES_ENTRY = "VariableManager:InternalVariables" # internal to current process (can store lambda) def __init__(self, sheerka): - super().__init__(sheerka) + super().__init__(sheerka, order=3) self.bound_variables = { - self.sheerka.name: {"enable_process_return_values", "save_execution_context"} + self.sheerka.name: {"enable_process_return_values", "save_execution_context", "enable_process_rules"} } def initialize(self): diff --git a/src/core/sheerka/services/sheerka_service.py b/src/core/sheerka/services/sheerka_service.py index 5db8096..c620792 100644 --- a/src/core/sheerka/services/sheerka_service.py +++ b/src/core/sheerka/services/sheerka_service.py @@ -1,6 +1,6 @@ from dataclasses import dataclass -from core.global_symbols import NotFound +from core.global_symbols import NotFound, ErrorObj from core.utils import sheerka_deepcopy @@ -14,8 +14,9 @@ class BaseService: Base class for services """ - def __init__(self, sheerka): + def __init__(self, sheerka, order=999): self.sheerka = sheerka + self.order = order # initialisation order. The lowest is initialized first def initialize(self): """ @@ -46,3 +47,8 @@ class BaseService: Store/record the value of an attribute """ self.sheerka.record_var(context, self.NAME, var_name, getattr(self, var_name)) + + +@dataclass() +class FailedToCompileError(Exception, ErrorObj): + cause: list diff --git a/src/core/tokenizer.py b/src/core/tokenizer.py index 1550e9d..f702965 100644 --- a/src/core/tokenizer.py +++ b/src/core/tokenizer.py @@ -139,6 +139,7 @@ class LexerError(Exception): class Keywords(Enum): DEF = "def" CONCEPT = "concept" + RULE = "rule" FROM = "from" BNF = "bnf" AS = "as" @@ -149,6 +150,7 @@ class Keywords(Enum): RET = "ret" WHEN = "when" PRINT = "print" + THEN = "then" class Tokenizer: @@ -557,36 +559,3 @@ class IterParser: return token_after except StopIteration: return Token(TokenKind.EOF, -1, -1, -1, -1) - - -# @dataclass -# class PropDef: -# prop: str -# index: int -# -# -# class SimpleExpressionParser(IterParser): -# def __init__(self, source): -# super().__init__(source) -# self.properties = [] -# -# def parse(self): -# -# prop, index, key = None, None, None -# while self.next_token(): -# if self.token.type == TokenKind.DOT: -# self.properties.append(PropDef(prop, index, key)) -# prop, index, key = None, None, None -# continue -# -# if self.token.type == TokenKind.LBRACKET: -# index = self.parse_index() -# elif self.token.type == TokenKind.LBRACE: -# key = self.parse_key() -# else: -# prop = self.token.value -# -# if prop is not None: -# self.properties.append(PropDef(prop, index, key)) -# -# def parse_i diff --git a/src/core/utils.py b/src/core/utils.py index 39cdd10..65fdf16 100644 --- a/src/core/utils.py +++ b/src/core/utils.py @@ -3,12 +3,14 @@ import importlib import inspect import os import pkgutil +import re from copy import deepcopy -from pyparsing import * +# from pyparsing import * +from pyparsing import Literal, Word, nums, Combine, Optional, delimitedList, oneOf, alphas, Suppress from core.global_symbols import CustomType -from core.tokenizer import TokenKind, Tokenizer +from core.tokenizer import TokenKind, Tokenizer, Token COLORS = { "black", @@ -249,7 +251,7 @@ def make_unique(lst, get_id=None): return list(_make_unique(lst, get_id)) -def product(a, b): +def sheerka_product(a, b): """ Kind of cartesian product between lists a and b knowing that a is also a list : a is a list of list !!! @@ -569,7 +571,7 @@ def as_bag(obj, forced_properties=None): """ Get the properties of an object (static and dynamic) :param obj: - :param forced_properties: + :param forced_properties: special mode where properties are given in parameter :return: """ @@ -638,6 +640,33 @@ def get_text_from_tokens(tokens, custom_switcher=None, tracker=None): return res +def tokens_are_matching(tokens1, tokens2, skip_tokens=True): + def get_next(it): + try: + return next(it) + except StopIteration: + return Token(TokenKind.EOF, "", -1, -1, -1) + + iter1 = iter(tokens1) + iter2 = iter(tokens2) + + while True: + t1 = get_next(iter1) + t2 = get_next(iter2) + + if skip_tokens: + if t1.type == TokenKind.WHITESPACE: + t1 = next(iter1) + if t2.type == TokenKind.WHITESPACE: + t2 = next(iter2) + + if t1.type == TokenKind.EOF and t2.type == TokenKind.EOF: + return True + + if t1.type != t2.type or t1.value != t2.value: + return False + + def dump_ast(node): dump = ast.dump(node) for to_remove in [", ctx=Load()", ", kind=None", ", type_ignores=[]"]: diff --git a/src/evaluators/DefConceptEvaluator.py b/src/evaluators/DefConceptEvaluator.py index 3170f4c..1d4ae76 100644 --- a/src/evaluators/DefConceptEvaluator.py +++ b/src/evaluators/DefConceptEvaluator.py @@ -2,13 +2,13 @@ import core.utils from core.ast_helpers import UnreferencedVariablesVisitor from core.builtin_concepts import ParserResultConcept, ReturnValueConcept, BuiltinConcepts from core.concept import Concept, DEFINITION_TYPE_BNF, DEFINITION_TYPE_DEF +from core.global_symbols import NotInit from core.sheerka.services.SheerkaExecute import ParserInput from core.tokenizer import TokenKind, Tokenizer from evaluators.BaseEvaluator import OneReturnValueEvaluator -from parsers.BaseParser import NotInitializedNode from parsers.BnfNodeParser import ParsingExpression, ParsingExpressionVisitor from parsers.DefConceptParser import DefConceptNode, NameNode -from parsers.PythonParser import PythonNode, get_python_node +from parsers.PythonParser import get_python_node class ConceptOrRuleNameVisitor(ParsingExpressionVisitor): @@ -70,7 +70,7 @@ class DefConceptEvaluator(OneReturnValueEvaluator): part_ret_val = getattr(def_concept_node, prop) # put back the sources - if isinstance(part_ret_val, NotInitializedNode): + if part_ret_val is NotInit: continue elif isinstance(part_ret_val, NameNode): source = str(part_ret_val) @@ -107,7 +107,7 @@ class DefConceptEvaluator(OneReturnValueEvaluator): concept.init_key(key_source) # update the bnf definition if needed - if not isinstance(def_concept_node.definition, NotInitializedNode) and \ + if def_concept_node.definition is not NotInit and \ def_concept_node.definition_type == DEFINITION_TYPE_BNF: concept.set_bnf(def_concept_node.definition.value.value) diff --git a/src/evaluators/DefRuleEvaluator.py b/src/evaluators/DefRuleEvaluator.py new file mode 100644 index 0000000..8e6a9cd --- /dev/null +++ b/src/evaluators/DefRuleEvaluator.py @@ -0,0 +1,56 @@ +import core.utils +from core.builtin_concepts import BuiltinConcepts, ParserResultConcept +from core.global_symbols import NotInit +from core.rule import Rule, ACTION_TYPE_PRINT, ACTION_TYPE_EXEC +from core.tokenizer import Keywords +from evaluators.BaseEvaluator import OneReturnValueEvaluator +from parsers.DefRuleParser import DefRuleNode, DefFormatRuleNode + + +class DefRuleEvaluator(OneReturnValueEvaluator): + """ + Used to store a new format rule + """ + NAME = "DefRule" + + def __init__(self): + super().__init__(self.NAME, [BuiltinConcepts.EVALUATION], 50) + + def matches(self, context, return_value): + return return_value.status and \ + isinstance(return_value.value, ParserResultConcept) and \ + isinstance(return_value.value.value, DefRuleNode) + + def eval(self, context, return_value): + """ + Creates a Rule out of a DefRule and saves it in db + :param context: + :param return_value: + :return: + """ + + context.log("Adding a new rule", self.name) + rule_definition = return_value.value.value + sheerka = context.sheerka + + name = None if rule_definition.name is NotInit else str(rule_definition.name) + predicate = core.utils.get_text_from_tokens(rule_definition.tokens[Keywords.WHEN][1:]) + if isinstance(rule_definition, DefFormatRuleNode): + action_type = ACTION_TYPE_PRINT + action = core.utils.get_text_from_tokens(rule_definition.tokens[Keywords.PRINT][1:]) + compiled_action = rule_definition.print + else: + action_type = ACTION_TYPE_EXEC + action = core.utils.get_text_from_tokens(rule_definition.tokens[Keywords.THEN][1:]) + compiled_action = rule_definition.then + + rule = Rule(action_type, name, predicate, action) + rule.compiled_predicates = rule_definition.when + rule.rete_disjunctions = rule_definition.rete + rule.compiled_action = compiled_action + + ret = sheerka.create_new_rule(context, rule) + if not ret.status: + error_cause = sheerka.objvalue(ret.body) + context.log(f"Failed to add new rule '{rule}'. Reason: {error_cause}", self.name) + return sheerka.ret(self.name, ret.status, ret.value, parents=[return_value]) diff --git a/src/evaluators/FormatRuleEvaluator.py b/src/evaluators/FormatRuleEvaluator.py deleted file mode 100644 index ea66dbb..0000000 --- a/src/evaluators/FormatRuleEvaluator.py +++ /dev/null @@ -1,45 +0,0 @@ -import core.utils -from core.builtin_concepts import BuiltinConcepts, ParserResultConcept -from core.rule import Rule -from core.tokenizer import Keywords -from evaluators.BaseEvaluator import OneReturnValueEvaluator -from parsers.DefFormatRuleParser import FormatRuleNode - - -class FormatRuleEvaluator(OneReturnValueEvaluator): - """ - Used to store a new format rule - """ - NAME = "FormatRule" - - def __init__(self): - super().__init__(self.NAME, [BuiltinConcepts.EVALUATION], 50) - - def matches(self, context, return_value): - return return_value.status and \ - isinstance(return_value.value, ParserResultConcept) and \ - isinstance(return_value.value.value, FormatRuleNode) - - def eval(self, context, return_value): - """ - Creates a Rule out of a FormatRuleNode and saves it in db - :param context: - :param return_value: - :return: - """ - - context.log("Adding a new format rule", self.name) - format_rule_node = return_value.value.value - sheerka = context.sheerka - - predicate = core.utils.get_text_from_tokens(format_rule_node.tokens[Keywords.WHEN][1:]) - action = core.utils.get_text_from_tokens(format_rule_node.tokens[Keywords.PRINT][1:]) - rule = Rule("print", None, predicate, action) - rule.compiled_predicate = format_rule_node.rule - rule.compiled_action = format_rule_node.format_ast - - ret = sheerka.create_new_rule(context, rule) - if not ret.status: - error_cause = sheerka.objvalue(ret.body) - context.log(f"Failed to add new rule '{rule}'. Reason: {error_cause}", self.name) - return sheerka.ret(self.name, ret.status, ret.value, parents=[return_value]) diff --git a/src/evaluators/PythonEvaluator.py b/src/evaluators/PythonEvaluator.py index 4be43c4..d8f9ddb 100644 --- a/src/evaluators/PythonEvaluator.py +++ b/src/evaluators/PythonEvaluator.py @@ -36,12 +36,16 @@ def inject_context(context): class Expando: - def __init__(self, bag): + def __init__(self, name, bag): + self.__name = name for k, v in bag.items(): setattr(self, k, v) def __repr__(self): - return f"{dir(self)}" + return f"{vars(self)}" + + def get_name(self): + return self.__name @dataclass @@ -89,12 +93,12 @@ class PythonEvaluator(OneReturnValueEvaluator): def eval(self, context, return_value): sheerka = context.sheerka - node = get_python_node(return_value.value.value) + node = return_value.value.value.get_python_node() debugger = context.get_debugger(PythonEvaluator.NAME, "eval") debugger.debug_entering(node=node) exception_debugger = context.get_debugger("Exceptions", PythonEvaluator.NAME + ".eval") - get_trace_back = context.debug_enabled or exception_debugger.is_enabled() + get_trace_back = exception_debugger.is_enabled() context.log(f"Evaluating python node {node}.", self.name) @@ -179,8 +183,6 @@ class PythonEvaluator(OneReturnValueEvaluator): """ unreferenced_names_visitor = UnreferencedNamesVisitor(context) names = unreferenced_names_visitor.get_names(node.ast_) - if context.debug_enabled: - context.debug(self.NAME, "eval", "names", names) return self.get_globals_by_names(context, names, node, expression_only) def get_sheerka_method(self, context, name, expression_only): @@ -200,6 +202,7 @@ class PythonEvaluator(OneReturnValueEvaluator): my_globals = { "Concept": core.concept.Concept, "BuiltinConcepts": core.builtin_concepts.BuiltinConcepts, + "Expando": Expando, "ExecutionContext": ExecutionContext, "in_context": context.in_context, } @@ -218,14 +221,14 @@ class PythonEvaluator(OneReturnValueEvaluator): continue # support reference to sheerka - if name == "sheerka": + if name.lower() == "sheerka": bag = {} visitor = NamesWithAttributesVisitor() for sequence in visitor.get_sequences(node.ast_, "sheerka"): if (len(sequence) > 1 and (method := self.get_sheerka_method(context, sequence[1], expression_only)) is not None): bag[sequence[1]] = method - my_globals["sheerka"] = Expando(bag) + my_globals[name] = Expando("sheerka", bag) continue # search in local variables. To remove when local variables will be merged with memory diff --git a/src/out/DeveloperVisitor.py b/src/out/DeveloperVisitor.py index 152167e..e15ccf4 100644 --- a/src/out/DeveloperVisitor.py +++ b/src/out/DeveloperVisitor.py @@ -62,7 +62,8 @@ class DeveloperVisitor: return format_ast.clone(value=res) except NameError as error: - context.debug("DeveloperVisitor", "visit_FormatAstList", "evaluate_expression", error, is_error=True) + self.debugger.debug_log(f"Error : {error}", is_error=True) + self.debugger.debug_var(f"Exception", error, is_error=True) return FormatAstVariableNotFound(format_ast.name) def visit_FormatAstSequence(self, context, format_ast, bag): diff --git a/src/parsers/BaseCustomGrammarParser.py b/src/parsers/BaseCustomGrammarParser.py index f794c68..ae45488 100644 --- a/src/parsers/BaseCustomGrammarParser.py +++ b/src/parsers/BaseCustomGrammarParser.py @@ -62,6 +62,38 @@ class KeywordNotFound(CustomGrammarParserNode, ParsingError): return hash(self.keywords) +@dataclass() +class NameNode(CustomGrammarParserNode): + + def get_name(self): + name = "" + first = True + for token in self.tokens: + if token.type == TokenKind.EOF: + break + if token.type == TokenKind.WHITESPACE: + continue + if not first: + name += " " + + name += token.value[1:-1] if token.type == TokenKind.STRING else str(token.value) + first = False + + return name + + def __repr__(self): + return self.get_name() + + def __eq__(self, other): + if not isinstance(other, NameNode): + return False + + return self.get_name() == other.get_name() + + def __hash__(self): + return hash(self.get_name()) + + class BaseCustomGrammarParser(BaseParser): """ Base class for sheerka specific grammars diff --git a/src/parsers/BaseNodeParser.py b/src/parsers/BaseNodeParser.py index 6bd13e6..838d127 100644 --- a/src/parsers/BaseNodeParser.py +++ b/src/parsers/BaseNodeParser.py @@ -42,6 +42,9 @@ class LexerNode(Node): def to_short_str(self): raise NotImplementedError + def get_source_to_parse(self): + return self.source + class UnrecognizedTokensNode(LexerNode): def __init__(self, start, end, tokens): @@ -296,6 +299,12 @@ class SourceCodeNode(LexerNode): def to_short_str(self): return f"SCN('{self.source}')" + def get_python_node(self): + return self.python_node + + def get_source_to_parse(self): + return self.python_node.source + class SourceCodeWithConceptNode(LexerNode): """ @@ -409,6 +418,12 @@ class SourceCodeWithConceptNode(LexerNode): def to_short_str(self): return f"SCWC({self.first}" + ", ".join(n.to_short_str for n in self.nodes) + f"{self.last})" + def get_python_node(self): + return self.python_node + + def get_source_to_parse(self): + return self.python_node.source + @dataclass() class GrammarErrorNode(ParsingError): diff --git a/src/parsers/BaseParser.py b/src/parsers/BaseParser.py index 59d6178..70e5abd 100644 --- a/src/parsers/BaseParser.py +++ b/src/parsers/BaseParser.py @@ -1,33 +1,12 @@ -import logging from dataclasses import dataclass from typing import Union from core.builtin_concepts import BuiltinConcepts, ParserResultConcept from core.concept import Concept -from core.error import ErrorObj +from core.global_symbols import ErrorObj from core.sheerka.ExecutionContext import ExecutionContext from core.sheerka.services.SheerkaExecute import ParserInput -from core.sheerka_logger import get_logger -from core.tokenizer import TokenKind, Token, Tokenizer, LexerError - - -# # keep a cache for the parser input -# pi_cache = Cache(default=lambda key: ParserInput(key), max_size=20) -# -# -# def get_parser_input(text, tokens=None, length=None): -# """ -# Returns new or existing parser input -# :param text: -# :param tokens: -# :param length: -# :return: -# """ -# if tokens is None or pi_cache.has(text): -# return pi_cache.get(text) -# pi = ParserInput(text, tokens, length) -# pi_cache.put(text, pi) -# return pi +from core.tokenizer import TokenKind, Token, LexerError @dataclass() @@ -35,13 +14,6 @@ class Node: pass -class NotInitializedNode(Node): - pass - - def __repr__(self): - return "**N/A**" - - @dataclass() class ParsingError(Node, ErrorObj): pass @@ -206,17 +178,6 @@ class BaseParser: return parser_input.value - # @staticmethod - # def manage_eof(lst, strip_eof): - # if strip_eof: - # if len(lst) and lst[-1].type == TokenKind.EOF: - # lst.pop() - # return lst - # - # if len(lst) == 0 or not lst[-1].type == TokenKind.EOF: - # lst.append(Token(TokenKind.EOF, "", -1, -1, -1)) - # return lst - @staticmethod def get_tokens_boundaries(tokens): """ diff --git a/src/parsers/ComparisonParser.py b/src/parsers/ComparisonParser.py new file mode 100644 index 0000000..5d4b08b --- /dev/null +++ b/src/parsers/ComparisonParser.py @@ -0,0 +1,185 @@ +from typing import Union, List + +from core.builtin_concepts_ids import BuiltinConcepts +from core.sheerka.services.SheerkaExecute import ParserInput, SheerkaExecute +from core.tokenizer import TokenKind, Token +from core.utils import get_text_from_tokens +from parsers.BaseParser import BaseParser +from parsers.expressions import ComparisonNode, ParenthesisMismatchError, NameExprNode, ComparisonType, VariableNode + + +class ComparisonParser(BaseParser): + """ + Parses xxx (== | > | < | >= | <= | != | in | not in) yyy + Nothing else + """ + + def __init__(self, **kwargs): + super().__init__("Expression", 60, False, yield_eof=True) + + def parse(self, context, parser_input: Union[ParserInput, List[Token]]): + """ + :param context: + :param parser_input: + :return: + """ + + if isinstance(parser_input, list): + parser_input = context.sheerka.services[SheerkaExecute.NAME].get_parser_input(None, tokens=parser_input) + + elif not isinstance(parser_input, ParserInput): + return None + + context.log(f"Parsing '{parser_input}' with ComparisonExpressionParser", self.name) + sheerka = context.sheerka + + if parser_input.is_empty(): + return context.sheerka.ret(self.name, + False, + sheerka.new(BuiltinConcepts.IS_EMPTY)) + + if not self.reset_parser(context, parser_input): + return self.sheerka.ret( + self.name, + False, + context.sheerka.new(BuiltinConcepts.ERROR, body=self.error_sink)) + + self.parser_input.next_token() + + node = self.parse_compare() + + value = self.get_return_value_body(context.sheerka, self.parser_input.as_text(), node, node) + + ret = self.sheerka.ret( + self.name, + not self.has_error, + value) + + return ret + + def parse_compare(self): + start = self.parser_input.pos + left = self.parse_names() + if left is None: + return None + + if (comp := self.eat_comparison()) is None: + return left + + right = self.parse_names() + end = right.end if right else self.parser_input.pos + return ComparisonNode(start, end, self.parser_input.tokens[start: end + 1], comp, left, right) + + def parse_names(self): + + token = self.parser_input.token + if token.type == TokenKind.EOF: + return None + + buffer = [] + paren_count = 0 + last_lparen = None + last_rparen = None + start = self.parser_input.pos + while (paren_count > 0 or not self.eat_comparison(False)) and token.type != TokenKind.EOF: + buffer.append(token) + if token.type == TokenKind.LPAR: + last_lparen = token + paren_count += 1 + if token.type == TokenKind.RPAR: + last_rparen = token + paren_count -= 1 + self.parser_input.next_token(False) + token = self.parser_input.token + + if paren_count != 0: + pass + + if paren_count > 0: + self.error_sink.append(ParenthesisMismatchError(last_lparen)) + return None + + if paren_count < 0: + self.error_sink.append(ParenthesisMismatchError(last_rparen)) + return None + + if buffer[-1].type == TokenKind.WHITESPACE: + buffer.pop() + + end = start + len(buffer) - 1 + return self.try_to_recognize(NameExprNode(start, end, buffer)) + + def eat_comparison(self, eat=True): + token = self.parser_input.token + if token.type == TokenKind.EQUALSEQUALS: + if eat: + self.parser_input.next_token() + return ComparisonType.EQUALS + + if token.type == TokenKind.LESS: + if self.parser_input.the_token_after(False).type == TokenKind.EQUALS: + if eat: + self.parser_input.next_token() + self.parser_input.next_token() + return ComparisonType.LESS_THAN_OR_EQUALS + else: + if eat: + self.parser_input.next_token() + return ComparisonType.LESS_THAN + + if token.type == TokenKind.GREATER: + if self.parser_input.the_token_after(False).type == TokenKind.EQUALS: + if eat: + self.parser_input.next_token() + self.parser_input.next_token() + return ComparisonType.GREATER_THAN_OR_EQUALS + else: + if eat: + self.parser_input.next_token() + return ComparisonType.GREATER_THAN + + if token.type == TokenKind.IDENTIFIER and token.value == "not": + if self.parser_input.the_token_after(True).value == "in": + if eat: + self.parser_input.next_token() + self.parser_input.next_token() + return ComparisonType.NOT_IN + + if token.type == TokenKind.IDENTIFIER and token.value == "in": + if eat: + self.parser_input.next_token() + return ComparisonType.IN + + if token.type == TokenKind.EMARK and self.parser_input.the_token_after(False).type == TokenKind.EQUALS: + if eat: + self.parser_input.next_token() + self.parser_input.next_token() + return ComparisonType.NOT_EQUAlS + + return None + + @staticmethod + def try_to_recognize(expr: NameExprNode): + not_a_variable = False + expect_dot = False + for t in expr.tokens: + if expect_dot and t.type != TokenKind.DOT: + not_a_variable = True + if t.type == TokenKind.DOT: + break # Only interested in the root part + elif t.type == TokenKind.WHITESPACE: + expect_dot = True + elif t.type == TokenKind.LPAR: + pass # try to recognize function + elif not str(t.value).isidentifier(): + not_a_variable = True + + if not_a_variable: + return expr + + full_name = get_text_from_tokens(expr.tokens) + split = full_name.split(".") + if len(split) == 1: + return VariableNode(expr.start, expr.end, expr.tokens, split[0]) + else: + return VariableNode(expr.start, expr.end, expr.tokens, split[0], *split[1:]) diff --git a/src/parsers/DefConceptParser.py b/src/parsers/DefConceptParser.py index 69162a3..39fc183 100644 --- a/src/parsers/DefConceptParser.py +++ b/src/parsers/DefConceptParser.py @@ -1,13 +1,14 @@ -from dataclasses import dataclass, field +from dataclasses import dataclass import core.builtin_helpers import core.utils -from core.builtin_concepts import BuiltinConcepts, ReturnValueConcept, ParserResultConcept -from core.concept import ConceptParts, DEFINITION_TYPE_BNF, DEFINITION_TYPE_DEF +from core.builtin_concepts import BuiltinConcepts, ReturnValueConcept +from core.concept import DEFINITION_TYPE_BNF, DEFINITION_TYPE_DEF +from core.global_symbols import NotInit from core.sheerka.services.SheerkaExecute import ParserInput, SheerkaExecute from core.tokenizer import TokenKind, Keywords -from parsers.BaseCustomGrammarParser import BaseCustomGrammarParser, SyntaxErrorNode -from parsers.BaseParser import Node, ParsingError, NotInitializedNode, UnexpectedTokenParsingError +from parsers.BaseCustomGrammarParser import BaseCustomGrammarParser, SyntaxErrorNode, NameNode, CustomGrammarParserNode +from parsers.BaseParser import ParsingError, UnexpectedTokenParsingError from parsers.BnfDefinitionParser import BnfDefinitionParser @@ -17,15 +18,7 @@ class ParsingException(Exception): @dataclass() -class DefConceptParsingResult(Node): - """ - Base node for all default parser nodes - """ - tokens: list = field(compare=False, repr=False) - - -@dataclass() -class DefConceptParsingError(DefConceptParsingResult, ParsingError): +class DefConceptParsingError(CustomGrammarParserNode, ParsingError): pass @@ -38,63 +31,21 @@ class CannotHandleParsingError(DefConceptParsingError): @dataclass() -class NameNode(DefConceptParsingResult): - - def get_name(self): - name = "" - first = True - for token in self.tokens: - if token.type == TokenKind.EOF: - break - if token.type == TokenKind.WHITESPACE: - continue - if not first: - name += " " - - name += token.value[1:-1] if token.type == TokenKind.STRING else str(token.value) - first = False - - return name - - def __repr__(self): - return self.get_name() - - def __eq__(self, other): - if not isinstance(other, NameNode): - return False - - return self.get_name() == other.get_name() - - def __hash__(self): - return hash(self.get_name()) - - -@dataclass() -class DefConceptNode(DefConceptParsingResult): - name: NameNode = NotInitializedNode() - where: ReturnValueConcept = NotInitializedNode() - pre: ReturnValueConcept = NotInitializedNode() - post: ReturnValueConcept = NotInitializedNode() - body: ReturnValueConcept = NotInitializedNode() - ret: ReturnValueConcept = NotInitializedNode() - definition: ReturnValueConcept = NotInitializedNode() +class DefConceptNode(CustomGrammarParserNode): + name: NameNode = NotInit + where: ReturnValueConcept = NotInit + pre: ReturnValueConcept = NotInit + post: ReturnValueConcept = NotInit + body: ReturnValueConcept = NotInit + ret: ReturnValueConcept = NotInit + definition: ReturnValueConcept = NotInit definition_type: str = None - def get_asts(self): - asts = {} - for part_key in ConceptParts: - prop_value = getattr(self, part_key.value) - if isinstance(prop_value, ReturnValueConcept) and \ - isinstance(prop_value.body, ParserResultConcept) and \ - hasattr(prop_value.body.body, "ast_"): - asts[part_key] = prop_value - return asts - @dataclass() -class IsaConceptNode(DefConceptParsingResult): - concept: NameNode = NotInitializedNode() - set: NameNode = NotInitializedNode() +class IsaConceptNode(CustomGrammarParserNode): + concept: NameNode = NotInit + set: NameNode = NotInit class DefConceptParser(BaseCustomGrammarParser): @@ -201,12 +152,12 @@ class DefConceptParser(BaseCustomGrammarParser): def get_concept_definition(self, current_concept_def, parts): if Keywords.FROM not in parts: - return None, NotInitializedNode() + return None, NotInit tokens = parts[Keywords.FROM] if len(tokens) == 1: self.add_error(SyntaxErrorNode([], f"Empty '{tokens[0].value}' declaration."), False) - return None, NotInitializedNode() + return None, NotInit if tokens[1].value == Keywords.BNF.value: return self.get_concept_bnf_definition(current_concept_def, core.utils.strip_tokens(tokens[2:])) @@ -216,7 +167,7 @@ class DefConceptParser(BaseCustomGrammarParser): def get_concept_bnf_definition(self, current_concept_def, tokens): if len(tokens) == 0: self.add_error(SyntaxErrorNode([], "Empty 'bnf' declaration"), False) - return None, NotInitializedNode() + return None, NotInit if tokens[0].type == TokenKind.COLON: tokens = self.get_body(tokens[1:]) @@ -233,7 +184,7 @@ class DefConceptParser(BaseCustomGrammarParser): if not parsing_result.status: self.add_error(parsing_result.value) - return None, NotInitializedNode() + return None, NotInit return DEFINITION_TYPE_BNF, parsing_result @@ -243,7 +194,7 @@ class DefConceptParser(BaseCustomGrammarParser): tokens = core.utils.strip_tokens(tokens[start:]) if len(tokens) == 0: self.add_error(SyntaxErrorNode([], f"Empty 'from' declaration."), False) - return None, NotInitializedNode() + return None, NotInit if tokens[0].type == TokenKind.COLON: tokens = self.get_body(tokens[1:]) @@ -252,7 +203,7 @@ class DefConceptParser(BaseCustomGrammarParser): def get_ast(self, keyword, parts): if keyword not in parts: - return NotInitializedNode() + return NotInit tokens = parts[keyword] if len(tokens) == 1: diff --git a/src/parsers/DefFormatRuleParser.py b/src/parsers/DefFormatRuleParser.py deleted file mode 100644 index 1c0f62e..0000000 --- a/src/parsers/DefFormatRuleParser.py +++ /dev/null @@ -1,122 +0,0 @@ -from dataclasses import dataclass - -import core.utils -from core.builtin_concepts import BuiltinConcepts, ReturnValueConcept -from core.sheerka.services.SheerkaExecute import ParserInput -from core.sheerka.services.SheerkaRuleManager import SheerkaRuleManager, FormatAstNode -from core.tokenizer import Keywords -from parsers.BaseCustomGrammarParser import BaseCustomGrammarParser, KeywordNotFound -from parsers.BaseParser import BaseParser, Node - - -@dataclass -class FormatRuleNode(Node): - tokens: dict - rule: ReturnValueConcept = None - format_ast: FormatAstNode = None - - -class DefFormatRuleParser(BaseCustomGrammarParser): - """ - Class that will parse formatting rules definitions - eg: when xxx print yyy - where xxx will be evaluated in the context of BuiltinConcepts.EVAL_QUESTION_REQUESTED - and yyy is a internal way to describe a format (yet another one) - """ - - KEYWORDS = [Keywords.WHEN, Keywords.PRINT] - KEYWORDS_VALUES = [k.value for k in KEYWORDS] - - def __init__(self, **kwargs): - BaseCustomGrammarParser.__init__(self, "DefFormatRule", 60) - - def parse(self, context, parser_input: ParserInput): - """ - - :param context: - :param parser_input: - :return: - """ - - if not isinstance(parser_input, ParserInput): - return None - - if parser_input.from_tokens: - ret = context.sheerka.ret( - self.name, - False, - context.sheerka.new(BuiltinConcepts.NOT_FOR_ME, body=parser_input)) - self.log_result(context, parser_input, ret) - return ret - - context.log(f"Parsing '{parser_input}' with FunctionParser", self.name) - sheerka = context.sheerka - - if parser_input.is_empty(): - return sheerka.ret(self.name, - False, - sheerka.new(BuiltinConcepts.IS_EMPTY)) - - if not self.reset_parser(context, parser_input): - return self.sheerka.ret(self.name, - False, - context.sheerka.new(BuiltinConcepts.ERROR, body=self.error_sink)) - - self.parser_input.next_token() - rule = self.parse_rule() - body = self.get_return_value_body(sheerka, parser_input.as_text(), rule, rule) - ret = sheerka.ret(self.name, not self.has_error, body) - - self.log_result(context, parser_input.as_text(), ret) - return ret - - def parse_rule(self): - parts = self.get_parts(self.KEYWORDS_VALUES, strip_tokens=True) - if parts is None: - return None - - node = FormatRuleNode(parts) - try: - res = self.get_when(parts[Keywords.WHEN]) - if res is None: - return node - node.rule = res - - parsed = self.get_print(parts[Keywords.PRINT]) - if parsed is None: - return node - node.format_ast = parsed - except KeyError as e: - self.add_error(KeywordNotFound([], [e.args[0].value])) - return None - - return node - - def get_when(self, tokens): - """ - Validate the when part of the rule. - :param tokens: - :return: - """ - source = core.utils.get_text_from_tokens(core.utils.strip_tokens(tokens[1:])) - res = self.sheerka.services[SheerkaRuleManager.NAME].compile_when(self.context, self.name, source) - - if not isinstance(res, list): - self.add_error(res.value) - return None - - return res - - def get_print(self, tokens): - """ - Validate the print part - :param tokens: - :return: - """ - source = core.utils.get_text_from_tokens(core.utils.strip_tokens(tokens[1:])) - res = self.sheerka.services[SheerkaRuleManager.NAME].compile_print(self.context, source) - if not res.status: - self.add_error(res.value) - return None - - return res.body diff --git a/src/parsers/DefRuleParser.py b/src/parsers/DefRuleParser.py new file mode 100644 index 0000000..ac4516f --- /dev/null +++ b/src/parsers/DefRuleParser.py @@ -0,0 +1,231 @@ +from dataclasses import dataclass +from typing import List + +import core.utils +from core.builtin_concepts import ReturnValueConcept +from core.builtin_concepts_ids import BuiltinConcepts +from core.global_symbols import NotInit +from core.sheerka.services.SheerkaExecute import ParserInput +from core.sheerka.services.SheerkaRuleManager import SheerkaRuleManager, FormatAstNode, RuleCompiledPredicate +from core.sheerka.services.sheerka_service import FailedToCompileError +from core.tokenizer import Keywords, TokenKind +from parsers.BaseCustomGrammarParser import BaseCustomGrammarParser, NameNode, KeywordNotFound, SyntaxErrorNode +from parsers.BaseParser import Node, UnexpectedEofParsingError +from sheerkarete.conditions import Condition + + +@dataclass() +class DefRuleNode(Node): + tokens: dict + name: NameNode = NotInit + when: List[RuleCompiledPredicate] = NotInit + rete: List[List[Condition]] = NotInit + + +@dataclass() +class DefExecRuleNode(DefRuleNode): + then: ReturnValueConcept = NotInit + + +@dataclass +class DefFormatRuleNode(DefRuleNode): + print: FormatAstNode = NotInit + + +class DefRuleParser(BaseCustomGrammarParser): + DEF_KEYWORDS = [Keywords.RULE, Keywords.AS] + DEF_KEYWORDS_VALUES = [k.value for k in DEF_KEYWORDS] + + RULE_KEYWORDS = [Keywords.WHEN, Keywords.THEN, Keywords.PRINT] + RULE_KEYWORDS_VALUES = [k.value for k in RULE_KEYWORDS] + + def __init__(self, **kwargs): + BaseCustomGrammarParser.__init__(self, "DefRule", 60) + + def parse(self, context, parser_input: ParserInput): + if not isinstance(parser_input, ParserInput): + return None + + # rule parser can only manage string text + if parser_input.from_tokens: + ret = context.sheerka.ret( + self.name, + False, + context.sheerka.new(BuiltinConcepts.NOT_FOR_ME, body=parser_input)) + self.log_result(context, parser_input, ret) + return ret + + context.log(f"Parsing '{parser_input}' with DefRuleParser", self.name) + sheerka = context.sheerka + + if parser_input.is_empty(): + return sheerka.ret(self.name, + False, + sheerka.new(BuiltinConcepts.IS_EMPTY)) + + if not self.reset_parser(context, parser_input): + return self.sheerka.ret(self.name, + False, + context.sheerka.new(BuiltinConcepts.ERROR, body=self.error_sink)) + + self.parser_input.next_token() + node = self.parse_def_rule() + + body = self.get_return_value_body(sheerka, parser_input.as_text(), node, node) + ret = sheerka.ret(self.name, not self.has_error, body) + + self.log_result(context, parser_input.as_text(), ret) + return ret + + def parse_def_rule(self): + token = self.parser_input.token + if token.value == Keywords.DEF.value: + return self.parse_rule_name() + elif token.value in (Keywords.WHEN.value, Keywords.PRINT.value): + return self.parse_rule() + else: + self.add_error(KeywordNotFound([], [Keywords.WHEN.value])) + return None + + def parse_rule_name(self): + """ + Parses def rule xxx as yyyy + """ + self.parser_input.next_token() # eat def + token = self.parser_input.token + if token.value != Keywords.RULE.value: + self.add_error(KeywordNotFound([token], [Keywords.RULE.value])) + return None + + buffer = [] + while self.parser_input.next_token(skip_whitespace=False): + token = self.parser_input.token + if token.value == Keywords.AS.value: + break + else: + buffer.append(token) + else: # 'as' keyword not found + self.add_error(KeywordNotFound([], [Keywords.AS.value])) + return None + + if not self.parser_input.next_token(): # eat as + self.add_error(UnexpectedEofParsingError("While parsing 'when'.")) + return None + + rule = self.parse_rule() + + name_node = self.get_concept_name(buffer) + if name_node is None: + return rule + + rule.name = name_node + return rule + + def parse_rule(self): + """" + Parses 'when xxx then yyy' + or 'when xxx print yyy' + """ + parts = self.get_parts(self.RULE_KEYWORDS_VALUES, strip_tokens=True) + if Keywords.THEN in parts and Keywords.PRINT in parts: + self.add_error(SyntaxErrorNode([], "Cannot have both 'print' and 'then' keywords")) + return None + + if Keywords.THEN not in parts and Keywords.PRINT not in parts: + self.add_error(KeywordNotFound([], [Keywords.THEN.value, Keywords.PRINT.value])) + return None + + return self.parse_format_rule(parts) if Keywords.PRINT in parts else self.parse_exec_rule(parts) + + def parse_exec_rule(self, parts): + node = DefExecRuleNode(parts) + try: + compiled_result = self.get_when(parts[Keywords.WHEN]) + if compiled_result is None: + return node + node.when = compiled_result.compiled_predicates + node.rete = compiled_result.rete_disjunctions + + parsed = self.get_then(parts[Keywords.THEN]) + if parsed is None: + return node + node.then = parsed + except KeyError as e: + self.add_error(KeywordNotFound([], [e.args[0].value])) + return None + + return node + + def parse_format_rule(self, parts): + node = DefFormatRuleNode(parts) + try: + compiled_result = self.get_when(parts[Keywords.WHEN]) + if compiled_result is None: + return node + node.when = compiled_result.compiled_predicates + node.rete = compiled_result.rete_disjunctions + + parsed = self.get_print(parts[Keywords.PRINT]) + if parsed is None: + return node + node.print = parsed + except KeyError as e: + self.add_error(KeywordNotFound([], [e.args[0].value])) + return None + + return node + + def get_when(self, tokens): + """ + Validate the when part of the rule. + :param tokens: + :return: + """ + source = core.utils.get_text_from_tokens(core.utils.strip_tokens(tokens[1:])) + try: + rule_manager_service = self.sheerka.services[SheerkaRuleManager.NAME] + compiled_result = rule_manager_service.compile_when(self.context, self.name, source) + except FailedToCompileError as ex: + for c in ex.cause: + self.add_error(c) + return None + + return compiled_result + + def get_then(self, tokens): + source = core.utils.get_text_from_tokens(core.utils.strip_tokens(tokens[1:])) + res = self.sheerka.services[SheerkaRuleManager.NAME].compile_exec(self.context, source) + + if not res.status: + self.add_error(res.value) + return None + + return res + + def get_print(self, tokens): + """ + Validate the print part + :param tokens: + :return: + """ + source = core.utils.get_text_from_tokens(core.utils.strip_tokens(tokens[1:])) + res = self.sheerka.services[SheerkaRuleManager.NAME].compile_print(self.context, source) + if not res.status: + self.add_error(res.value) + return None + + return res.body + + def get_concept_name(self, tokens): + name_tokens = core.utils.strip_tokens(tokens) + if len(name_tokens) == 0: + self.add_error(SyntaxErrorNode([], "Name is mandatory")) + return None + + for token in name_tokens: + if token.type == TokenKind.NEWLINE: + self.add_error(SyntaxErrorNode([token], "Newline are not allowed in name.")) + return None + + name_node = NameNode(name_tokens) # skip the first token + return name_node diff --git a/src/parsers/ExpressionParser.py b/src/parsers/ExpressionParser.py index 77c026d..201e9c9 100644 --- a/src/parsers/ExpressionParser.py +++ b/src/parsers/ExpressionParser.py @@ -1,183 +1,64 @@ -from dataclasses import dataclass -from typing import List, Tuple, Callable +from itertools import product from core.builtin_concepts import BuiltinConcepts -from core.concept import Concept +from core.builtin_helpers import only_successful, parse_unrecognized, get_inner_body, parse_python, \ + get_lexer_nodes_using_positions from core.sheerka.services.SheerkaExecute import ParserInput -from core.tokenizer import TokenKind, Token -from parsers.BaseParser import Node, BaseParser, UnexpectedTokenParsingError, UnexpectedEofParsingError, ParsingError +from core.sheerka.services.sheerka_service import FailedToCompileError +from core.tokenizer import TokenKind, Tokenizer, Keywords +from core.utils import get_text_from_tokens +from parsers.BaseNodeParser import UnrecognizedTokensNode +from parsers.BaseParser import BaseParser, UnexpectedTokenParsingError, UnexpectedEofParsingError +from parsers.PythonWithConceptsParser import PythonWithConceptsParser +from parsers.expressions import ParenthesisNode, OrNode, AndNode, NotNode, LeftPartNotFoundError, \ + ParenthesisMismatchError, NameExprNode, ExprNode, VariableNode, ComparisonNode +from sheerkarete.common import V +from sheerkarete.conditions import Condition, AndConditions -class ExprNode(Node): - """ - Base ExprNode - eval() must be overridden - """ +class ReteConditionsEmitter: - def eval(self, obj): - return True + def __init__(self, context): + from parsers.ComparisonParser import ComparisonParser + self.context = context + self.comparison_parser = ComparisonParser() + self.var_counter = 0 + self.variables = {} + def add_variable(self, target): + var_name = f"__x_{self.var_counter:02}__" + self.var_counter += 1 + self.variables[target] = var_name + return var_name -@dataclass() -class LeftPartNotFoundError(ParsingError): - """ - When the expression starts with 'or' or 'and' - """ - pass + def init_variable_if_needed(self, node, res): + if node.name not in self.variables: + var_name = self.add_variable(node.name) + res.append(Condition(V(var_name), "__name__", node.name)) + return V(self.variables[node.name]) -class NameExprNode(ExprNode): - def __init__(self, tokens): - self.tokens = tokens - self.value = "".join([t.str_value for t in self.tokens]) + def get_conditions(self, expr_nodes): + conditions = [] + for expr_node in expr_nodes: + parsed_ret = self.comparison_parser.parse(self.context, expr_node.tokens) + if not parsed_ret.status: + raise FailedToCompileError(parsed_ret.body) + tree = parsed_ret.body.body - def eval(self, obj): - return self.value + if isinstance(tree, VariableNode): + var_name = self.init_variable_if_needed(tree, conditions) + if tree.attributes_str is not None: + conditions.append(Condition(var_name, tree.attributes_str, True)) - def __repr__(self): - return f"NameExprNode('{self.value}')" + elif isinstance(tree, ComparisonNode): + if isinstance(tree.left, VariableNode): + left = self.init_variable_if_needed(tree.left, conditions) + attr = tree.left.attributes_str or "__self__" + right = eval(get_text_from_tokens(tree.right.tokens)) + conditions.append(Condition(left, attr, right)) - def __str__(self): - return self.value - - -@dataclass -class PropertyEqualsNode(ExprNode): - prop: str - value: object - - def eval(self, obj): - if hasattr(obj, self.prop): - return str(getattr(obj, self.prop)) == self.value - - return False - - -@dataclass() -class PropertyContainsNode(ExprNode): - prop: str - value: object - - def eval(self, obj): - if hasattr(obj, self.prop): - return self.value in str(getattr(obj, self.prop)) - - return False - - -@dataclass -class PropertyEqualsSequenceNode(ExprNode): - """ - To use when the test must be done across parent and child - """ - props: List[str] - values: List[object] - - def eval(self, obj): - index = len(self.props) - 1 - - while True: - if not hasattr(obj, self.props[index]) or getattr(obj, self.props[index]) != self.values[index]: - return False - - if index == 0: - break - - index -= 1 - obj = obj.get_parent() if hasattr(obj, "get_parent") else obj.parent - if obj is None: - return False - - return True - - -@dataclass() -class IsaNode(ExprNode): - """ - To use to replicate instanceof, sheerka.instanceof, - """ - obj_class: object - - def eval(self, obj): - if isinstance(self.obj_class, type): - return isinstance(obj, self.obj_class) - - if isinstance(self.obj_class, (BuiltinConcepts, str)): - return isinstance(obj, Concept) and str(self.obj_class) == obj.key - - return False - - -@dataclass() -class LambdaNode(ExprNode): - """ - Generic expression to ease the tests - """ - lambda_exp: Callable[[object], bool] - - def eval(self, obj): - try: - return self.lambda_exp(obj) - except Exception: - pass - - -@dataclass(init=False) -class AndNode(ExprNode): - parts: Tuple[ExprNode] - - def __init__(self, *parts: ExprNode): - self.parts = parts - - def eval(self, obj): - res = self.parts[0].eval(obj) and self.parts[1].eval(obj) - for part in self.parts[2:]: - res &= part.eval(obj) - return res - - def __repr__(self): - return f"AndNode(" + ", ".join([repr(p) for p in self.parts]) + ")" - - def __str__(self): - return " and ".join([str(p) for p in self.parts]) - - -@dataclass(init=False) -class OrNode(ExprNode): - parts: Tuple[ExprNode] - - def __init__(self, *parts: ExprNode): - self.parts = parts - - def eval(self, obj): - res = self.parts[0].eval(obj) or self.parts[1].eval(obj) - for part in self.parts[2:]: - res |= part.eval(obj) - return res - - def __repr__(self): - return f"OrNode(" + ", ".join([repr(p) for p in self.parts]) + ")" - - def __str__(self): - return " or ".join([str(p) for p in self.parts]) - - -@dataclass() -class NotNode(ExprNode): - node: ExprNode - - def eval(self, obj): - return not self.node.eval(obj) - - -class FalseNode(ExprNode): - def eval(self, obj): - return False - - -class TrueNode(ExprNode): - def eval(self, obj): - return True + return [AndConditions(conditions)] class ExpressionParser(BaseParser): @@ -191,6 +72,15 @@ class ExpressionParser(BaseParser): def __init__(self, **kwargs): super().__init__("Expression", 50, False, yield_eof=True) + self.and_tokens = list(Tokenizer(" and ", yield_eof=False)) + self.and_not_tokens = list(Tokenizer(" and not ", yield_eof=False)) + self.not_tokens = list(Tokenizer("not ", yield_eof=False)) + + @staticmethod + def clean_parenthesis_nodes(nodes): + for i, node in enumerate(nodes): + if isinstance(node, ParenthesisNode): + nodes[i] = node.node def parse(self, context, parser_input: ParserInput): """ @@ -232,6 +122,7 @@ class ExpressionParser(BaseParser): return ret def parse_or(self): + start = self.parser_input.pos expr = self.parse_and() token = self.parser_input.token if token.type != TokenKind.IDENTIFIER or token.value != "or": @@ -243,14 +134,19 @@ class ExpressionParser(BaseParser): expr = self.parse_and() if expr is None: self.add_error(UnexpectedEofParsingError("When parsing 'or'")) - return OrNode(*parts) + end = self.parser_input.pos + self.clean_parenthesis_nodes(parts) + return OrNode(start, end, self.parser_input.tokens[start: end + 1], *parts) parts.append(expr) token = self.parser_input.token - return OrNode(*parts) + end = parts[-1].end + self.clean_parenthesis_nodes(parts) + return OrNode(start, end, self.parser_input.tokens[start: end + 1], *parts) def parse_and(self): - expr = self.parse_names() + start = self.parser_input.pos + expr = self.parse_not() token = self.parser_input.token if token.type != TokenKind.IDENTIFIER or token.value != "and": return expr @@ -258,27 +154,46 @@ class ExpressionParser(BaseParser): parts = [expr] while token.type == TokenKind.IDENTIFIER and token.value == "and": self.parser_input.next_token() - expr = self.parse_names() + expr = self.parse_not() if expr is None: self.add_error(UnexpectedEofParsingError("When parsing 'and'")) - return AndNode(*parts) + end = self.parser_input.pos + self.clean_parenthesis_nodes(parts) + return AndNode(start, end, self.parser_input.tokens[start: end + 1], *parts) parts.append(expr) token = self.parser_input.token - return AndNode(*parts) + end = parts[-1].end + self.clean_parenthesis_nodes(parts) + return AndNode(start, end, self.parser_input.tokens[start: end + 1], *parts) + + def parse_not(self): + token = self.parser_input.token + start = self.parser_input.pos + if token.type == TokenKind.IDENTIFIER and token.value == "not": + self.parser_input.next_token() + parsed = self.parse_not() + node = parsed.node if isinstance(parsed, ParenthesisNode) else parsed + return NotNode(start, + parsed.end, + self.parser_input.tokens[start: parsed.end + 1], + node) + else: + return self.parse_names() def parse_names(self): def stop(): return token.type == TokenKind.EOF or \ paren_count == 0 and token.type == TokenKind.RPAR or \ - token.type == TokenKind.IDENTIFIER and token.value in ("and", "or") + token.type == TokenKind.IDENTIFIER and token.value in ("and", "or", "not") token = self.parser_input.token if token.type == TokenKind.EOF: return None if token.type == TokenKind.LPAR: + start = self.parser_input.pos self.parser_input.next_token() expr = self.parse_or() token = self.parser_input.token @@ -286,14 +201,18 @@ class ExpressionParser(BaseParser): self.error_sink.append( UnexpectedTokenParsingError(f"Unexpected token '{token}'", token, [TokenKind.RPAR])) return expr + end = self.parser_input.pos self.parser_input.next_token() - return expr + return ParenthesisNode(start, end, None, expr) buffer = [] paren_count = 0 + last_paren = None + start = self.parser_input.pos while not stop(): buffer.append(token) if token.type == TokenKind.LPAR: + last_paren = token paren_count += 1 if token.type == TokenKind.RPAR: paren_count -= 1 @@ -305,65 +224,106 @@ class ExpressionParser(BaseParser): self.error_sink.append(LeftPartNotFoundError()) return None + if paren_count != 0: + self.error_sink.append(ParenthesisMismatchError(last_paren)) + return None + if buffer[-1].type == TokenKind.WHITESPACE: buffer.pop() - return NameExprNode(buffer) + end = start + len(buffer) - 1 + return NameExprNode(start, end, buffer) + def compile_conjunctions(self, context, conjunctions, who): + """ + Transform a list of conjunctions (AND and OR) into one or multiple CompiledExpr + :param context: + :param conjunctions: list of ExprNode + :param who: service that calls the method + :returns: List Of CompiledExpr + May throw FailedToRecognized if a conjunction cannot be parsed + """ + recognized = [] + for conjunction in conjunctions: + # try to recognize conjunction, one by one + # negative conjunction can be a concept starting with 'not' + parsed_ret = parse_unrecognized( + context, + conjunction.get_value(), # we remove the 'NOT' part when needed to ease the recognition + parsers="all", + who=who, + prop=Keywords.WHEN, + filter_func=only_successful) -class ExpressionVisitor: - """ - Pyhtonic implementation of visitors for ExprNode - """ + if parsed_ret.status: + recognized.append(get_inner_body(context, parsed_ret.body)) + else: + raise FailedToCompileError(parsed_ret.body) - def visit(self, expr_node): - name = expr_node.__class__.__name__ + # for each conjunction, we have a list of recognized concepts (or python node) + # we need a cartesian product of the results + # Explanation for later + # conjunction[0] : 'x is a y' that can be resolved with two concepts c:|1001: and c:|1002: + # conjunction[1] : 'y is an z' that can also be resolved with two concepts (c:|1003: and c:|1004) + # so to understand the full question 'x is a y and y is an z' + # we can have c:|1001: then c:|1003: + # or c:|1001: then c:|1004: + # or c:|1002: then c:|1003: + # or c:|1002: then c:|1004: + # if one of this combination works, it means that the question 'x is a y and y is an z' was matched + # hence the cartesian product + product_of_recognized = list(product(*recognized)) - method = 'visit_' + name - visitor = getattr(self, method, self.generic_visit) - return visitor(expr_node) + return_values = [] + for recognized_conjunctions in product_of_recognized: + if len(recognized_conjunctions) == 1 and not isinstance(conjunctions[0], NotNode): + return_values.append(recognized_conjunctions[0]) + elif len(recognized_conjunctions) == 1 and recognized_conjunctions[0].who == "parsers.Python": + # it is a negated python Node. Need to parse again + ret = parse_python(context, source=str(conjunctions[0])) + if ret.status: + return_values.append(ret) + else: + # find a way to track the failure + pass + else: + # complex result. Use PythonWithNode + lexer_nodes = get_lexer_nodes_using_positions(recognized_conjunctions, + self._get_positions(conjunctions)) - def generic_visit(self, expr_node): - """Called if no explicit visitor function exists for a node.""" - for field, value in expr_node.__dict__.items(): - if isinstance(value, (list, tuple)): - for item in value: - if isinstance(item, ExprNode): - self.visit(item) - elif isinstance(value, ExprNode): - self.visit(value) + # put back the 'and' / 'not' node + for i in range(len(lexer_nodes) - 1, 0, -1): + end = lexer_nodes[i].start - 1 + start = lexer_nodes[i - 1].end + 1 + if isinstance(conjunctions[i], NotNode): + lexer_nodes.insert(i, UnrecognizedTokensNode(start, end, self.and_not_tokens)) + else: + lexer_nodes.insert(i, UnrecognizedTokensNode(start, end, self.and_tokens)) + # add the starting 'not' if needed + # and reindex the following positions + if isinstance(conjunctions[0], NotNode): + lexer_nodes[0].start = 2 + lexer_nodes.insert(0, UnrecognizedTokensNode(0, 1, self.not_tokens)) -class TrueifyVisitor(ExpressionVisitor): - """ - Visit an ExprNode - replace all the nodes containing a variable to 'trueify' with True - The node containing both variables to trueify and to skip are skipped - """ + python_with_concept_node_ret = PythonWithConceptsParser().parse_nodes(context, lexer_nodes) + if not python_with_concept_node_ret.status: + # find a way to track the failure + pass + return_values.append(python_with_concept_node_ret) - def __init__(self, to_trueify, to_skip): - self.to_trueify = to_trueify - self.to_skip = to_skip + rete_cond_emitter = ReteConditionsEmitter(context) + rete_disjunctions = rete_cond_emitter.get_conditions(conjunctions) - def visit_AndNode(self, expr_node): - parts = [] - for part in expr_node.parts: - parts.append(self.visit(part)) - return AndNode(*parts) + return return_values, rete_disjunctions - def visit_OrNode(self, expr_node): - parts = [] - for part in expr_node.parts: - parts.append(self.visit(part)) - return OrNode(*parts) - - def visit_NameExprNode(self, expr_node): - return_true = False - for t in expr_node.tokens: - if t.type == TokenKind.IDENTIFIER: - if t.value in self.to_skip: - return expr_node - if t.value in self.to_trueify: - return_true = True - - return NameExprNode([Token(TokenKind.IDENTIFIER, "True", -1, -1, -1)]) if return_true else expr_node + @staticmethod + def _get_positions(expr_nodes): + """ + simply manage NotNodes to address the fact that the 'not' part in removed + """ + for expr in expr_nodes: + if isinstance(expr, NotNode): + yield ExprNode(expr.start + 2, expr.end, expr.tokens[2:]) + else: + yield expr diff --git a/src/parsers/FunctionParser.py b/src/parsers/FunctionParser.py index 561f9d2..25e276b 100644 --- a/src/parsers/FunctionParser.py +++ b/src/parsers/FunctionParser.py @@ -44,9 +44,15 @@ class NamesNode(FunctionParserNode): return "".join([t.str_value for t in self.tokens]) def to_unrecognized(self): + """ + UnrecognizedTokensNode with all tokens + """ return UnrecognizedTokensNode(self.start, self.end, self.tokens).fix_source() def to_str_unrecognized(self): + """ + UnrecognizedTokensNode with one token, which is a string token of all the tokens + """ token = Token(TokenKind.STRING, "'" + self.str_value() + "'", self.tokens[0].index, @@ -342,6 +348,7 @@ class FunctionParser(BaseParser): res = [SourceCodeWithConceptNode(function_node.first.to_unrecognized(), function_node.last.to_unrecognized())] + # try to recognize every parameter, one by one for param in function_node.parameters: if isinstance(param.value, NamesNode): # try to recognize concepts diff --git a/src/parsers/PythonParser.py b/src/parsers/PythonParser.py index 7b378da..6262356 100644 --- a/src/parsers/PythonParser.py +++ b/src/parsers/PythonParser.py @@ -35,8 +35,9 @@ class ConceptDetectedError(ParsingError): class PythonNode(Node): - def __init__(self, source, ast_=None, objects=None): - self.source = source + def __init__(self, source, ast_=None, original_source=None, objects=None): + self.source = source # what was parsed + self.original_source = original_source or source # to remember source before concept id replacement self.ast_ = ast_ # if ast_ else ast.parse(source, mode="eval") if source else None self.objects = objects or {} # when objects (mainly concepts or rules) are recognized in the expression self.compiled = None @@ -64,6 +65,9 @@ class PythonNode(Node): if self.source != other.source: return False + if self.original_source != other.original_source: + return False + if self.ast_ and other.ast_: self_dump = self.get_dump(self.ast_) other_dump = self.get_dump(other.ast_) @@ -74,6 +78,9 @@ class PythonNode(Node): def __hash__(self): return hash((self.source, self.ast_.hash)) + def get_python_node(self): + return self + @staticmethod def get_dump(ast_): if not ast_: @@ -156,7 +163,7 @@ class PythonParser(BaseParser): BuiltinConcepts.PARSER_RESULT, parser=self, source=parser_input.as_text(), - body=PythonNode(source_code, tree, tracker), + body=PythonNode(source_code, tree, objects=tracker), try_parsed=None)) self.log_result(context, parser_input.as_text(), ret) diff --git a/src/parsers/PythonWithConceptsParser.py b/src/parsers/PythonWithConceptsParser.py index 2a4033b..8a7be3b 100644 --- a/src/parsers/PythonWithConceptsParser.py +++ b/src/parsers/PythonWithConceptsParser.py @@ -1,9 +1,8 @@ from core.builtin_concepts import BuiltinConcepts -from core.sheerka.services.SheerkaExecute import SheerkaExecute +from core.builtin_helpers import parse_python, CreateObjectIdentifiers from parsers.BaseNodeParser import ConceptNode, RuleNode from parsers.BaseNodeParser import SourceCodeWithConceptNode from parsers.BaseParser import BaseParser -from parsers.PythonParser import PythonParser from parsers.UnrecognizedNodeParser import UnrecognizedNodeParser unrecognized_nodes_parser = UnrecognizedNodeParser() @@ -13,16 +12,6 @@ class PythonWithConceptsParser(BaseParser): def __init__(self, **kwargs): super().__init__("PythonWithConcepts", 20) - @staticmethod - def sanitize(identifier): - if identifier is None: - return "" - - res = "" - for c in identifier: - res += c if c.isalnum() else "0" - return res - @staticmethod def get_nodes(nodes): for node in nodes: @@ -46,73 +35,47 @@ class PythonWithConceptsParser(BaseParser): source = "" to_parse = "" - identifiers = {} - identifiers_key = {} python_ids_mappings = {} - - def _get_identifier(c, wrapper): - """ - Get an identifier for a concept. - Make sure to return the same identifier if the same concept - Make sure to return a different identifier if same name but different concept - - Internal function because I don't want identifiers, identifiers_key and python_ids_mappings - to be instance variables - I would like to keep this parser as stateless as possible - :param c: - :return: - """ - if id(c) in identifiers: - return identifiers[id(c)] - - identifier = wrapper + self.sanitize(c.key or c.name) - if c.id: - identifier += "__" + c.id - - if identifier in identifiers_key: - identifiers_key[identifier] += 1 - identifier += f"_{identifiers_key[identifier]}" - else: - identifiers_key[identifier] = 0 - - identifier += wrapper - - identifiers[id(c)] = identifier - return identifier + last_token_index = 0 + ids_manager = CreateObjectIdentifiers() for node in self.get_nodes(nodes): if isinstance(node, ConceptNode): - source += node.source - if to_parse: + if node.start != last_token_index + 1 and source: # put back missing whitespace + source += " " to_parse += " " + + source += node.source concept = node.concept - python_id = _get_identifier(concept, "__C__") + python_id = ids_manager.get_identifier(concept, "__C__") to_parse += python_id python_ids_mappings[python_id] = concept + last_token_index = node.end elif isinstance(node, RuleNode): - source += node.source - if to_parse: + if node.start != last_token_index + 1 and source: # put back missing whitespace + source += " " to_parse += " " + + source += node.source rule = node.rule - python_id = _get_identifier(rule, "__R__") + python_id = ids_manager.get_identifier(rule, "__R__") to_parse += python_id python_ids_mappings[python_id] = rule + last_token_index = node.end else: source += node.source - to_parse += node.source + to_parse += node.get_source_to_parse() + last_token_index = node.end + if hasattr(node, "get_python_node"): + python_ids_mappings.update(node.get_python_node().objects) - with context.push(BuiltinConcepts.PARSE_CODE, - {"language": "Python", "source": to_parse}, - "Trying Python for '" + to_parse + "'") as sub_context: - parser_input = context.sheerka.services[SheerkaExecute.NAME].get_parser_input(to_parse) - python_parser = PythonParser() - result = python_parser.parse(sub_context, parser_input) + result = parse_python(context, to_parse, "Trying Python for '" + to_parse + "'") if result.status: python_node = result.body.body - python_node.source = source + python_node.original_source = source python_node.objects = python_ids_mappings return sheerka.ret( diff --git a/src/parsers/UnrecognizedNodeParser.py b/src/parsers/UnrecognizedNodeParser.py index 4f83181..44dd3ee 100644 --- a/src/parsers/UnrecognizedNodeParser.py +++ b/src/parsers/UnrecognizedNodeParser.py @@ -53,7 +53,7 @@ class UnrecognizedNodeParser(BaseParser): if not res.status: self.add_error(res.body) else: - sequences_found = core.utils.product(sequences_found, [res.body]) + sequences_found = core.utils.sheerka_product(sequences_found, [res.body]) elif isinstance(node, UnrecognizedTokensNode): res = parse_unrecognized(context, node.source, PARSERS) @@ -62,16 +62,16 @@ class UnrecognizedNodeParser(BaseParser): lexer_nodes = get_lexer_nodes(res.body.body, node.start, node.tokens) if lexer_nodes: # make lexer_nodes is not empty (for example, some Python result are discarded) - sequences_found = core.utils.product(sequences_found, lexer_nodes) + sequences_found = core.utils.sheerka_product(sequences_found, lexer_nodes) else: - sequences_found = core.utils.product(sequences_found, [node]) + sequences_found = core.utils.sheerka_product(sequences_found, [node]) has_unrecognized = True else: - sequences_found = core.utils.product(sequences_found, [node]) + sequences_found = core.utils.sheerka_product(sequences_found, [node]) has_unrecognized = True elif isinstance(node, SourceCodeNode): - sequences_found = core.utils.product(sequences_found, [node]) + sequences_found = core.utils.sheerka_product(sequences_found, [node]) has_unrecognized = True # to let PythonWithConceptParser validate the code elif isinstance(node, SourceCodeWithConceptNode): @@ -82,7 +82,7 @@ class UnrecognizedNodeParser(BaseParser): break else: node.nodes[i] = res.body - sequences_found = core.utils.product(sequences_found, [node]) + sequences_found = core.utils.sheerka_product(sequences_found, [node]) has_unrecognized = True # to let PythonWithConceptParser validate the code else: # cannot happen as of today :-) diff --git a/src/parsers/expressions.py b/src/parsers/expressions.py new file mode 100644 index 0000000..52a2ed4 --- /dev/null +++ b/src/parsers/expressions.py @@ -0,0 +1,414 @@ +from dataclasses import dataclass +from typing import List, Tuple + +from core.tokenizer import Token, TokenKind, Tokenizer +from core.utils import tokens_are_matching +from parsers.BaseParser import Node, ParsingError + + +class ComparisonType: + EQUALS = "EQ" + NOT_EQUAlS = "NOT_EQ" + LESS_THAN = "LT" + LESS_THAN_OR_EQUALS = "LTE" + GREATER_THAN = "GT" + GREATER_THAN_OR_EQUALS = "GTE" + IN = "IN" + NOT_IN = "NOT_IN" + + +@dataclass() +class LeftPartNotFoundError(ParsingError): + """ + When the expression starts with 'or' or 'and' + """ + pass + + +@dataclass() +class ParenthesisMismatchError(ParsingError): + token: Token + + +@dataclass +class ExprNode(Node): + """ + Base ExprNode + eval() must be overridden + """ + start: int # index of the first token + end: int # index of the last token + tokens: List[Token] + + def eval(self, obj): + return True + + def __eq__(self, other): + if not isinstance(other, ExprNode): + return False + + if self.start != other.start or self.end != other.end: + return False + + if other.tokens is not None and other.tokens != self.tokens: + return False + + return True + + def __hash__(self): + return hash((self.start, self.end)) + + +class NameExprNode(ExprNode): + def __init__(self, start, end, tokens): + super().__init__(start, end, tokens) + self.tokens = tokens + self.value = "".join([t.str_value for t in self.tokens]) + + def eval(self, obj): + return self.value + + def get_value(self): + return self.value + + def __repr__(self): + return f"NameExprNode('{self.value}')" + + def __str__(self): + return self.value + + def __eq__(self, other): + if not isinstance(other, NameExprNode): + return False + + return super().__eq__(other) + + def __hash__(self): + return super().__hash__() + + +@dataclass(init=False) +class AndNode(ExprNode): + parts: Tuple[ExprNode] + + def __init__(self, start, end, tokens, *parts: ExprNode): + super().__init__(start, end, tokens) + self.parts = parts + + def eval(self, obj): + res = self.parts[0].eval(obj) and self.parts[1].eval(obj) + for part in self.parts[2:]: + res &= part.eval(obj) + return res + + def __repr__(self): + return f"AndNode(start={self.start}, end={self.end}, " + ", ".join([repr(p) for p in self.parts]) + ")" + + def __str__(self): + return " and ".join([str(p) for p in self.parts]) + + def __eq__(self, other): + if not isinstance(other, AndNode): + return False + + if self.start != other.start or self.end != other.end: + return False + + if other.tokens is not None and other.tokens != self.tokens: + return False + + return self.parts == other.parts + + def __hash__(self): + return hash((self.start, self.end, self.parts)) + + +@dataclass(init=False) +class OrNode(ExprNode): + parts: Tuple[ExprNode] + + def __init__(self, start, end, tokens, *parts: ExprNode): + super().__init__(start, end, tokens) + self.parts = parts + + def eval(self, obj): + res = self.parts[0].eval(obj) or self.parts[1].eval(obj) + for part in self.parts[2:]: + res |= part.eval(obj) + return res + + def __repr__(self): + return f"OrNode(start={self.start}, end={self.end}, " + ", ".join([repr(p) for p in self.parts]) + ")" + + def __str__(self): + return " or ".join([str(p) for p in self.parts]) + + def __eq__(self, other): + if not isinstance(other, OrNode): + return False + + if self.start != other.start or self.end != other.end: + return False + + if other.tokens is not None and other.tokens != self.tokens: + return False + + return self.parts == other.parts + + def __hash__(self): + return hash((self.start, self.end, self.parts)) + + +@dataclass() +class NotNode(ExprNode): + node: ExprNode + + def eval(self, obj): + return not self.node.eval(obj) + + def get_value(self): + return self.node.get_value() + + def __repr__(self): + return f"NotNode(start={self.start}, end={self.end}, {self.node!r})" + + def __str__(self): + return f"not {self.node}" + + def __eq__(self, other): + if not isinstance(other, NotNode): + return False + + if self.start != other.start or self.end != other.end: + return False + + if other.tokens is not None and other.tokens != self.tokens: + return False + + return self.node == other.node + + def __hash__(self): + return hash((self.start, self.end, self.node)) + + +@dataclass() +class ParenthesisNode(ExprNode): + """ + Contains the boundaries of an expression inside parenthesis + Need it, just to keep track of the boundaries of the parenthesis + """ + node: ExprNode + + def __eq__(self, other): + if not isinstance(other, ParenthesisNode): + return False + + if self.start != other.start or self.end != other.end: + return False + + if other.tokens is not None and other.tokens != self.tokens: + return False + + return self.node == other.node + + def __hash__(self): + return hash((self.start, self.end, self.node)) + + +class VariableNode(ExprNode): + def __init__(self, start, end, tokens, name, *attributes): + super().__init__(start, end, tokens) + self.name = name.strip() + self.attributes = [attr.strip() for attr in attributes] + if len(self.attributes) > 0: + self.attributes_str = ".".join(self.attributes) + else: + self.attributes_str = None + + def __eq__(self, other): + if id(self) == id(other): + return True + + if not isinstance(other, VariableNode): + return False + + return self.name == other.name and self.attributes == other.attributes + + def __hash__(self): + return hash((self.name, self.attributes)) + + def __repr__(self): + prefix = f"VariableNode(start={self.start}, end={self.end}, '{self.name}" + if len(self.attributes) > 0: + return prefix + "." + ".".join(self.attributes) + "')" + else: + return prefix + "')" + + def __str__(self): + if self.attributes: + return self.name + "." + ".".join(self.attributes) + else: + return self.name + + +@dataclass +class ComparisonNode(ExprNode): + comp: str + left: ExprNode + right: ExprNode + + def __eq__(self, other): + if id(self) == id(other): + return True + + if not isinstance(other, ComparisonNode): + return False + + return (self.comp == other.comp and + self.left == other.left and + self.right == other.right) + + def __hash__(self): + return hash((self.comp, self.left, self.right)) + + def __repr__(self): + return f"ComparisonNode(start={self.start}, end={self.end}, {self.left!r} {self.comp} {self.right!r})" + + def __str__(self): + return f"{self.left} {self.comp} {self.right}" + + +class ExpressionVisitor: + """ + Pyhtonic implementation of visitors for ExprNode + """ + + def visit(self, expr_node): + name = expr_node.__class__.__name__ + + method = 'visit_' + name + visitor = getattr(self, method, self.generic_visit) + return visitor(expr_node) + + def generic_visit(self, expr_node): + """Called if no explicit visitor function exists for a node.""" + for field, value in expr_node.__dict__.items(): + if isinstance(value, (list, tuple)): + for item in value: + if isinstance(item, ExprNode): + self.visit(item) + elif isinstance(value, ExprNode): + self.visit(value) + + +class TrueifyVisitor(ExpressionVisitor): + """ + Visit an ExprNode + replace all the nodes containing a variable to 'trueify' with True + The node containing both variables to trueify and to skip are skipped + """ + + def __init__(self, to_trueify, to_skip): + self.to_trueify = to_trueify + self.to_skip = to_skip + + def visit_AndNode(self, expr_node): + parts = [] + for part in expr_node.parts: + parts.append(self.visit(part)) + return AndNode(expr_node.start, expr_node.end, expr_node.tokens, *parts) + + def visit_OrNode(self, expr_node): + parts = [] + for part in expr_node.parts: + parts.append(self.visit(part)) + return OrNode(expr_node.start, expr_node.end, expr_node.tokens, *parts) + + def visit_NameExprNode(self, expr_node): + return_true = False + for t in expr_node.tokens: + if t.type == TokenKind.IDENTIFIER: + if t.value in self.to_skip: + return expr_node + if t.value in self.to_trueify: + return_true = True + + return NameExprNode(expr_node.start, + expr_node.end, + [Token(TokenKind.IDENTIFIER, "True", -1, -1, -1)]) if return_true else expr_node + + +is_question_tokens = list(Tokenizer("is_question()")) +eval_question_requested_in_context = list(Tokenizer("context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)")) + + +class IsAQuestionVisitor(ExpressionVisitor): + """ + visit an expression and return True if is_question or context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED) + if found. + """ + + def visit_NameExprNode(self, expr_node): + if tokens_are_matching(expr_node.tokens, is_question_tokens) or \ + tokens_are_matching(expr_node.tokens, eval_question_requested_in_context): + return True + return None + + def visit_AndNode(self, expr_node): + """ + AND | True | False | None + ------+-------+-------+---------- + False | False | False | False + True | True | False | True + None | True | False | None + """ + res = self.visit(expr_node.parts[0]) + if isinstance(res, bool) and not res: + return res + + for part in expr_node.parts[1:]: + visited = self.visit(part) + if isinstance(visited, bool): + if not visited: + return visited + else: + res = visited + + return res + + def visit_OrNode(self, expr_node): + """ + OR | True | False | None + ------+-------+-------+---------- + True | True | True | True + False | True | False | False + None | True | False | None + """ + res = self.visit(expr_node.parts[0]) + if isinstance(res, bool) and res: + return res + + for part in expr_node.parts[1:]: + visited = self.visit(part) + if isinstance(visited, bool): + if visited: + return visited + else: + res = visited + + return res + + def visit_NotNode(self, expr_node): + """ + | NOT + ------+------- + False | True + True | False + None | None + """ + visited = self.visit(expr_node.node) + return None if visited is None else not visited + + def is_a_question(self, expr_node): + res = self.visit(expr_node) + return isinstance(res, bool) and res diff --git a/src/sdp/sheerkaDataProvider.py b/src/sdp/sheerkaDataProvider.py index 2c19bd2..5d084b0 100644 --- a/src/sdp/sheerkaDataProvider.py +++ b/src/sdp/sheerkaDataProvider.py @@ -567,7 +567,8 @@ class SheerkaDataProvider: text = self.io.read_text(ontology_file) return text.split("\n") - def test_only_destroy_refs(self): - current_sdp_refs_folder = self.io.path_join(self.RefFolder, self.name) + def test_only_destroy_refs(self, name=None): + name = name or self.name + current_sdp_refs_folder = self.io.path_join(self.RefFolder, name) if path.exists(current_sdp_refs_folder): shutil.rmtree(current_sdp_refs_folder) diff --git a/src/sheerkarete/__init__.py b/src/sheerkarete/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/sheerkarete/alpha.py b/src/sheerkarete/alpha.py new file mode 100644 index 0000000..0dd399f --- /dev/null +++ b/src/sheerkarete/alpha.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +if TYPE_CHECKING: # pragma: no cover + from typing import List + from sheerkarete.join_node import JoinNode + from sheerkarete.common import WME + + +class AlphaMemory: + + def __init__(self, key, condition, items=None, successors=None): + """ + Stores a set of WMEs (items). If activating an activated wme does not + exist, then it addes it. It also right activates all of its successors, + which correspond to beta nodes. + :param items: set of WMEs that match this memory + :param successors: list of JoinNode + """ + self.key = key + self.condition = condition + self.items: List[WME] = items if items else [] + self.successors: List[JoinNode] = successors if successors else [] + self.reference_count = 0 + + def activation(self, wme: WME) -> None: + """ + Adds the WME to the alpha memory and then right activates the children + in the beta network. Note, these are activated in reversed order to + prevent duplicate matches. + """ + if not self.condition.test(wme): + return + + self.items.append(wme) + wme.amems.append(self) + for child in reversed(self.successors): + child.right_activation(wme) diff --git a/src/sheerkarete/beta.py b/src/sheerkarete/beta.py new file mode 100644 index 0000000..56832bd --- /dev/null +++ b/src/sheerkarete/beta.py @@ -0,0 +1,55 @@ +from __future__ import annotations +from typing import TYPE_CHECKING + +from sheerkarete.common import ReteToken + +if TYPE_CHECKING: # pragma: no cover + from typing import Any + from typing import List + from typing import Dict + from typing import Optional + from sheerkarete.common import V + from sheerkarete.common import WME + + +class ReteNode: + """ + Base BetaNode class, tracks parent and children. + """ + + def __init__(self, children=None, parent=None, **kwargs): + super().__init__(**kwargs) + self.children: List[ReteNode] = children if children else [] + self.parent: Optional[ReteNode] = parent + + def find_nearest_ancestor_with_same_amem(self, amem): + return None + + +class BetaMemory(ReteNode): + """ + A memory node for the beta network. Contains items (tokens) and a list of + `all_children`, which is used in conjunction with `children` to support + left unlinking. + """ + + def __init__(self, items: Optional[List[ReteToken]] = None, **kwargs): + """ + Similar to alpha memory, but items is a set of tokens instead of wmes. + """ + super().__init__(**kwargs) + self.items: List[ReteToken] = items if items else [] + self.all_children: List[ReteNode] = [] + + def find_nearest_ancestor_with_same_amem(self, amem): + return self.parent.find_nearest_ancestor_with_same_amem(amem) + + def left_activation(self, token, wme: Optional[WME] = None, binding: Optional[Dict[V, Any]] = None): + """ + Creates a new token based on the incoming token/wme, adds it to the + memory (items) then activates the children with the token. + """ + new_token = ReteToken(token, wme, node=self, binding=binding) + self.items.append(new_token) + for child in self.children: + child.left_activation(new_token) diff --git a/src/sheerkarete/bind_node.py b/src/sheerkarete/bind_node.py new file mode 100644 index 0000000..969db56 --- /dev/null +++ b/src/sheerkarete/bind_node.py @@ -0,0 +1,60 @@ +from __future__ import annotations +from typing import TYPE_CHECKING +import inspect + +from sheerkarete.beta import ReteNode +from sheerkarete.common import V + +if TYPE_CHECKING: # pragma: no cover + from typing import Any + from typing import Dict + from sheerkarete.network import ReteNetwork + + +class BindNode(ReteNode): + """ + A beta network class. This class stores a code snipit, with variables in + it. It gets all the bindings from the incoming token, updates them with the + current bindings, binds the result to the target variable (to), then + activates its children with the updated bindings. + """ + + def __init__(self, children, parent, func, to, rete: ReteNetwork): + """ + :type children: + :type parent: BetaNode + :type to: str + """ + super().__init__(children=children, parent=parent) + self.func = func + self.func_args = inspect.getfullargspec(self.func)[0] + self.bind = to + self._rete_net = rete + + def get_function_result(self, binding: Dict[V, Any]): + """ + Given a binding that maps variables to values, this instantiates the + arguments for the function and executes it. + """ + args = {arg: self._rete_net if arg == 'network' else + self._rete_net.facts[binding[V(arg)]] if + binding[V(arg)] in self._rete_net.facts else + binding[V(arg)] for arg in self.func_args} + return self.func(**args) + + def left_activation(self, token, wme, binding): + """ + Copies and updates the bindings with the results of the function + execution. It then left_activates children with this binding. + """ + result = self.get_function_result(binding) + + if self.bind in binding: + if binding[self.bind] != result: + return + else: + binding = binding.copy() + binding[self.bind] = result + + for child in self.children: + child.left_activation(token, wme, binding) diff --git a/src/sheerkarete/common.py b/src/sheerkarete/common.py new file mode 100644 index 0000000..e8b07fd --- /dev/null +++ b/src/sheerkarete/common.py @@ -0,0 +1,201 @@ +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +if TYPE_CHECKING: # pragma: no cover + from typing import Hashable + from typing import List + from typing import Optional + from sheerkarete.alpha import AlphaMemory + from sheerkarete.pnode import PNode + + +@dataclass(eq=True, frozen=True) +class V: + """ + A variable for pattern matching. + """ + __slots__ = ['name'] + name: str + + def __repr__(self): + return "V({})".format(self.name) + + +@dataclass(eq=True, frozen=True) +class Match: + pnode: PNode + token: ReteToken + + def fire(self): + print(f"rule {self.pnode.rule} if fired with {self.token=}") + # return self.pnode.rule.fire(self.token) + + +class WME: + """ + This is essentially a fact, it has no variables in it. A working memory is + essentially comprised of a collection of these elements. + """ + __slots__ = ['identifier', 'attribute', 'value', 'amems', 'tokens', + 'negative_join_results'] + + def __init__(self, identifier: str, attribute: Hashable, value: Hashable) -> None: + """ + Identifier, attribute, and value can be any kind of object except V + objects (i.e., variables). + """ + if (isinstance(identifier, V) or isinstance(attribute, V) or + isinstance(value, V)): + raise ValueError("WMEs cannot have variables (V objects).") + + self.identifier = identifier + self.attribute = attribute + self.value = value + from sheerkarete.alpha import AlphaMemory + self.amems: List[AlphaMemory] = [] # the ones containing this WME + self.tokens: List[ReteToken] = [] # the ones containing this WME + self.negative_join_results: List[NegativeJoinResult] = [] + + def __hash__(self): + return hash((self.identifier, self.attribute, self.value)) + + def __repr__(self): + return "(%s ^%s %s)" % (self.identifier, self.attribute, self.value) + + def __eq__(self, other: object) -> bool: + """ + :type other: WME + """ + if not isinstance(other, WME): + return False + return self.identifier == other.identifier and \ + self.attribute == other.attribute and \ + self.value == other.value + + +class ReteToken: + """ + Tokens represent matches within the alpha and beta memories. The parent + corresponds to the match that was extended to create the current token. + """ + __slots__ = ['parent', 'wme', 'node', 'children', 'join_results', + 'ncc_results', 'owner', 'binding'] + + def __init__(self, parent, wme, node=None, binding=None): + """ + :type wme: WME + :type parent: Token + :type binding: dict + """ + self.parent = parent + self.wme = wme + # points to memory this token is in + self.node = node + # the ones with parent = this token + self.children: List[ReteToken] = [] + # used only on tokens in negative nodes + self.join_results: List[NegativeJoinResult] = [] + self.ncc_results: List[ReteToken] = [] + # Ncc + self.owner: Optional[ReteToken] = None + self.binding = binding if binding else {} # {"$x": "B1"} + + if self.parent: + self.parent.children.append(self) + if self.wme: + self.wme.tokens.append(self) + + def __repr__(self) -> str: + return "" % self.wmes + + def __eq__(self, other: object) -> bool: + return (isinstance(other, ReteToken) and self.parent == other.parent and + self.wme == other.wme) + + def is_root(self) -> bool: + return not self.parent and not self.wme + + @property + def wmes(self) -> List[Optional[WME]]: + ret = [self.wme] + t = self + while t.parent and not t.parent.is_root(): + t = t.parent + ret.insert(0, t.wme) + return ret + + def delete_descendents_of_token(self) -> None: + """ + Helper function to delete all the descendent tokens. + """ + for t in self.children: + t.delete_token_and_descendents() + + def delete_token_and_descendents(self) -> None: + """ + Deletes a token and its descendents, but has special cases that make + this difficult to understand in isolation. + + TODO: + - Add optimization for right unlinking (pg 87 of Doorenbois + thesis). + + :type token: Token + """ + from sheerkarete.ncc_node import NccNode + from sheerkarete.ncc_node import NccPartnerNode + from sheerkarete.negative_node import NegativeNode + from sheerkarete.beta import BetaMemory + from sheerkarete.join_node import JoinNode + + for child in self.children: + child.delete_token_and_descendents() + + if (isinstance(self.node, BetaMemory) and not + isinstance(self.node, NccPartnerNode)): + self.node.items.remove(self) + + if self.wme: + self.wme.tokens.remove(self) + if self.parent: + self.parent.children.remove(self) + + if isinstance(self.node, BetaMemory): + if not self.node.items: + for bmchild in self.node.children: + if (isinstance(bmchild, JoinNode) and bmchild in + bmchild.amem.successors): + bmchild.amem.successors.remove(bmchild) + + if isinstance(self.node, NegativeNode): + if not self.node.items: + self.node.amem.successors.remove(self.node) + for jr in self.join_results: + jr.wme.negative_join_results.remove(jr) + + if isinstance(self.node, NccNode): + for result_tok in self.ncc_results: + if result_tok.wme: + result_tok.wme.tokens.remove(result_tok) + if result_tok.parent: + result_tok.parent.children.remove(result_tok) + + elif isinstance(self.node, NccPartnerNode): + self.owner.ncc_results.remove(self) + if not self.owner.ncc_results and self.node.ncc_node: + for bchild in self.node.ncc_node.children: + bchild.left_activation(self.owner, None, + self.owner.binding) + + +@dataclass(eq=True, frozen=True) +class NegativeJoinResult: + """ + A new class to store the result of a negative join. Similar to a token, it + is owned by a token. + """ + __slots__ = ['owner', 'wme'] + owner: ReteToken + wme: WME diff --git a/src/sheerkarete/conditions.py b/src/sheerkarete/conditions.py new file mode 100644 index 0000000..f403226 --- /dev/null +++ b/src/sheerkarete/conditions.py @@ -0,0 +1,212 @@ +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING, Callable, Union + +from sheerkarete.common import V + +if TYPE_CHECKING: # pragma: no cover + from typing import List + from typing import Tuple + from typing import Hashable + from sheerkarete.common import WME + + +@dataclass(eq=True, frozen=True) +class AndConditions: + """ + List of Conditions semantically bound by and ANDs + To avoid manipulating list of list + """ + conditions: List[Union[Condition, Tuple]] + + +class ConditionalList(tuple): + """ + A conditional that consists of a list of other conditionals. + """ + + def __new__(cls, *args): + return super().__new__(cls, args) + + def __repr__(self): + return "{}{}".format(self.__class__.__name__, super().__repr__()) + + def __hash__(self): + return hash((self.__class__.__name__, tuple(self))) + + +@dataclass(eq=True, frozen=True) +class Condition: + """ + Triplet identifier attribute value to match + """ + __slots__ = ['identifier', 'attribute', 'value'] + identifier: Hashable + attribute: Hashable + value: Hashable + + def __repr__(self): + return "(%s ^%s %s)" % (self.identifier, self.attribute, self.value) + + def __str__(self): + return f"{self.identifier}.{self.attribute} = {self.value}" + + def get_key(self): + id_test = '*' + attr_test = '*' + value_test = '*' + + if not isinstance(self.identifier, V): + id_test = self.identifier + if not isinstance(self.attribute, V): + attr_test = self.attribute + if not isinstance(self.value, V): + value_test = self.value + + return id_test, attr_test, value_test + + @property + def vars(self) -> List[Tuple[str, V]]: + """ + Returns a list of tuples (field, var) that contains the slot names as a + string and the variable object it maps to. + """ + return [(field, getattr(self, field)) + for field in ('identifier', 'attribute', 'value') + if isinstance(getattr(self, field), V)] + + def contain(self, v: V) -> str: + """ + Checks if a variable is in the condition. Returns as string with the + name of the field if it is, otherwise an empty string. + """ + assert isinstance(v, V) + + for f in ['identifier', 'attribute', 'value']: + _v = getattr(self, f) + if _v == v: + return f + return "" + + def test(self, w: WME) -> bool: + """ + Checks if a pattern matches a working memory element. + """ + vars_values = {} + for f in ['identifier', 'attribute', 'value']: + self_value = getattr(self, f) + wme_value = getattr(w, f) + if isinstance(self_value, V): + if self_value in vars_values and vars_values[self_value] != wme_value: + return False + vars_values[self_value] = wme_value + elif self_value != wme_value: + return False + return True + + def __hash__(self): + return hash(('cond', self.identifier, self.attribute, self.value)) + + +class NotEqualsCondition(Condition): + """ + Test not equals of the value + """ + + def test(self, w: WME) -> bool: + """ + Checks if a pattern matches a working memory element. + """ + for f in ['identifier', 'attribute']: + v = getattr(self, f) + if isinstance(v, V): + continue + if v != getattr(w, f): + return False + + v = getattr(self, "value") + return isinstance(v, V) or getattr(w, "value") != v + + def get_key(self): + id_test = '*' + attr_test = '*' + + if not isinstance(self.identifier, V): + id_test = self.identifier + if not isinstance(self.attribute, V): + attr_test = self.attribute + + return id_test, attr_test, "*" + + def __hash__(self): + return hash(('ne-cond', self.identifier, self.attribute, self.value)) + + def __repr__(self): + return "(%s ^%s != %s)" % (self.identifier, self.attribute, self.value) + + def __str__(self): + return f"{self.identifier}.{self.attribute} != {self.value}" + + +class NegatedCondition(Condition): + """ + A negated pattern. + """ + + def __repr__(self): + return "-(%s ^%s %s)" % (self.identifier, self.attribute, self.value) + + def __hash__(self): + return hash(('neg', self.identifier, self.attribute, self.value)) + + +class NegatedConjunctiveConditions(ConditionalList): + """ + A negated conjunction of conditions. + """ + + def __repr__(self): + return "-{}".format(super(NegatedConjunctiveConditions, self).__repr__()) + + @property + def number_of_conditions(self) -> int: + return len(self) + + def __hash__(self): + return hash(('ncc', tuple(self))) + + +@dataclass(eq=True, frozen=True) +class FilterCondition: + """ + This is a test, it includes a function that might include variables. + When employed in rete, the variable bindings are passed in as keyword args + and the function is exectued. The function must return a boolean and if it + evaluates to true, then the condition matches otherwise it does not. + """ + __slots__ = ['func'] + func: Callable + + def __repr__(self): + return "Filter({})".format(repr(self.func)) + + def __hash__(self): + return hash(('filter', self.func)) + + +@dataclass(eq=True, frozen=True) +class BindCondition: + """ + Similar to Filter, but binds the result of a function execution to a new + variable. + """ + __slots__ = ['func', 'to'] + func: Callable + to: V + + def __repr__(self): + return "Bind({},{})".format(repr(self.func), repr(self.to)) + + def __hash__(self): + return hash(('bind', self.func, self.to)) diff --git a/src/sheerkarete/filter_node.py b/src/sheerkarete/filter_node.py new file mode 100644 index 0000000..e7a6ea2 --- /dev/null +++ b/src/sheerkarete/filter_node.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +import inspect +from typing import TYPE_CHECKING + +from sheerkarete.beta import ReteNode +from sheerkarete.common import V + +if TYPE_CHECKING: # pragma: no cover + from typing import List + from typing import Callable + from sheerkarete.network import ReteNetwork + + +class FilterNode(ReteNode): + """ + A beta network node. Takes a function, passes variables in as kwargs, and + executes it. If the code evaluates to True (boolean), then it activates the + children with the token/wme. + """ + + def __init__(self, children: List[ReteNode], + parent: ReteNode, + func: Callable, + rete: ReteNetwork): + super().__init__(children=children, parent=parent) + self.func = func + self.func_args = inspect.getfullargspec(self.func)[0] + self._rete_net = rete + + def get_function_result(self, token, wme, binding): + args = {arg: self._rete_net if arg == 'network' else + self._rete_net.facts[binding[V(arg)]] if + binding[V(arg)] in self._rete_net.facts else + binding[V(arg)] for arg in self.func_args} + + return self.func(**args) + + def left_activation(self, token, wme, binding): + """ + :type binding: dict + :type wme: WME + :type token: Token + """ + result = self.get_function_result(token, wme, binding) + if bool(result): + for child in self.children: + child.left_activation(token, wme, binding) diff --git a/src/sheerkarete/join_node.py b/src/sheerkarete/join_node.py new file mode 100644 index 0000000..0ddee74 --- /dev/null +++ b/src/sheerkarete/join_node.py @@ -0,0 +1,140 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Dict, Any + +from sheerkarete.alpha import AlphaMemory +from sheerkarete.beta import ReteNode +from sheerkarete.common import V, WME, ReteToken + +if TYPE_CHECKING: # pragma: no cover + from sheerkarete.conditions import Condition + + +class JoinNode(ReteNode): + """ + A beta network class. Does the heavly lifting of joining tokens from a beta + memory with wmes from an alpha memory. + + This class has an alpha memory connected to its right side, which triggers + right_activations. + + The parent constitutes the left side (another node in the beta network), + which triggers left_activations. + + When the JoinNode is right activated, it checks the incoming wme against + all the tokens in the parent (on the left side), using the tests. For + every match, updated bindings are created and the children are activated. + + When the JoinNode is left activated, it checks the incoming token against + the wmes from the alpha memory instead (essentially the opposite direction + as above). Similarly, for matches, updated bindings are created and + children are activated. + """ + + def __init__(self, amem: AlphaMemory, condition: Condition, **kwargs): + super().__init__(**kwargs) + self.amem: AlphaMemory = amem + self.condition = condition + self.nearest_ancestor_with_same_amem = None + self.vars = [(v, field) for field, v in self.condition.vars if + isinstance(v, V)] + + @property + def amem_recently_nonempty(self) -> bool: + return len(self.amem.items) == 1 + + @property + def parent_recently_nonempty(self) -> bool: + return len(self.parent.items) == 1 + + @property + def right_unlinked(self) -> bool: + return len(self.parent.items) == 0 + + @property + def left_unlinked(self) -> bool: + return len(self.amem.items) == 0 + + def update_nearest_ancestor_with_same_amem(self): + ancestor = self.parent.find_nearest_ancestor_with_same_amem(self.amem) + self.nearest_ancestor_with_same_amem = ancestor + + def find_nearest_ancestor_with_same_amem(self, amem: AlphaMemory): + if self.amem == amem: + return self + return self.parent.find_nearest_ancestor_with_same_amem(amem) + + def relink_to_alpha_memory(self): + """ + When there is not parent to the join, we remove the children (successor) or the alpha memory + This procedure put back the link between an amem and the join node + """ + ancestor = self.nearest_ancestor_with_same_amem + while ancestor and ancestor.right_unlinked: + ancestor = ancestor.nearest_ancestor_with_same_amem + if ancestor: + try: + loc = self.amem.successors.index(ancestor) + except ValueError: + loc = -1 + self.amem.successors.insert(loc + 1, self) + else: + self.amem.successors.insert(0, self) + + def relink_to_beta_memory(self): + self.parent.children.append(self) + + def right_activation(self, wme: WME) -> None: + """ + Called when an element is added to the respective alpha memory. + We look every tokens from the parent and check if it validates the condition brought by the amem + """ + if self.amem_recently_nonempty: + self.relink_to_beta_memory() + if not self.parent.items: + try: + self.amem.successors.remove(self) + except ValueError: + pass + for token in self.parent.items: + if self.perform_join_test(token, wme): + binding = self.make_binding(token, wme) + for child in self.children: + child.left_activation(token, wme, binding) + + def left_activation(self, token: ReteToken) -> None: + """ + Called when an element is added to the parent beta node. + """ + + if self.parent_recently_nonempty: + self.relink_to_alpha_memory() + if not self.amem.items: + self.parent.children.remove(self) + + for wme in self.amem.items: + if self.perform_join_test(token, wme): + binding = self.make_binding(token, wme) + for child in self.children: + child.left_activation(token, wme, binding=binding) + + def perform_join_test(self, token: ReteToken, wme: WME) -> bool: + """ + Test if the token and wme are compatible. + """ + for v, field in self.vars: + if v in token.binding and getattr(wme, field) != token.binding[v]: + return False + return True + + def make_binding(self, token: ReteToken, wme: WME) -> Dict[V, Any]: + """ + Makes updated bindings that result from joining token and wme. + """ + new_binding = {v: getattr(wme, field) for v, field in self.vars} + if new_binding: + binding = token.binding.copy() + binding.update(new_binding) + return binding + else: + return token.binding diff --git a/src/sheerkarete/ncc_node.py b/src/sheerkarete/ncc_node.py new file mode 100644 index 0000000..72ed063 --- /dev/null +++ b/src/sheerkarete/ncc_node.py @@ -0,0 +1,79 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from sheerkarete.alpha import AlphaMemory +from sheerkarete.beta import BetaMemory +from sheerkarete.common import ReteToken + +if TYPE_CHECKING: # pragma: no cover + from typing import Optional + from typing import List + from typing import Dict + from typing import Any + from sheerkarete.beta import ReteNode + from sheerkarete.common import V + from sheerkarete.common import WME + + +class NccNode(BetaMemory): + """ + A beta network class for negated conjunctive conditions (ncc). + + This has a memory of tokens (items) and a partner node. On left_activation + (from parent), the node adds results from its partner's result buffer to + the newly created token's ncc_results list, and sets the owner of the + result to the new token. If the new token does not have a any results in + the ncc_results list, then it activates all the children. + """ + + def __init__(self, partner: NccPartnerNode = None, **kwargs): + super().__init__(**kwargs) + self.partner = partner + + def find_nearest_ancestor_with_same_amem(self, amem: AlphaMemory): + return self.partner.parent.find_nearest_ancestor_with_same_amem(amem) + + def left_activation(self, token: ReteToken, wme: WME, binding: Dict[V, Any]): + new_token = ReteToken(token, wme, self, binding) + self.items.append(new_token) + for result in self.partner.new_result_buffer: + self.partner.new_result_buffer.remove(result) + new_token.ncc_results.append(result) + result.owner = new_token + if not new_token.ncc_results: + for child in self.children: + child.left_activation(new_token, None, binding) + + +class NccPartnerNode: + """ + The partner node for negated conjunctive conditions node. + + Takes the associated ncc node, the number of conditions, and a buffer of + any new results. + """ + def __init__(self, parent: Optional[ReteNode] = None, + ncc_node: Optional[NccNode] = None, + number_of_conditions: int = 0, + new_result_buffer: Optional[List[ReteToken]] = None): + self.parent = parent + self.ncc_node = ncc_node + self.number_of_conditions = number_of_conditions + self.new_result_buffer = new_result_buffer if new_result_buffer else [] + + def left_activation(self, token: ReteToken, wme: WME, binding: Dict[V, Any]): + new_result = ReteToken(token, wme, self, binding) + owners_t = token + owners_w = wme + for i in range(self.number_of_conditions): + owners_w = owners_t.wme + owners_t = owners_t.parent + for token in self.ncc_node.items: + if token.parent == owners_t and token.wme == owners_w: + token.ncc_results.append(new_result) + new_result.owner = token + token.delete_descendents_of_token() + break + else: + self.new_result_buffer.append(new_result) diff --git a/src/sheerkarete/negative_node.py b/src/sheerkarete/negative_node.py new file mode 100644 index 0000000..34748c6 --- /dev/null +++ b/src/sheerkarete/negative_node.py @@ -0,0 +1,59 @@ +from __future__ import annotations +from typing import TYPE_CHECKING + +from sheerkarete.alpha import AlphaMemory +from sheerkarete.beta import BetaMemory +from sheerkarete.join_node import JoinNode +from sheerkarete.common import ReteToken, NegativeJoinResult + +if TYPE_CHECKING: # pragma: no cover + from typing import Dict + from typing import Any + from sheerkarete.common import V + from sheerkarete.common import WME + + +class NegativeNode(BetaMemory, JoinNode): + """ + A beta network class that only passes on tokens when there is no match. The + left activation is called by the parent beta node. The right activation is + called from the alpha network (amem). Test are similar to those that + appear in JoinNode + """ + + def find_nearest_ancestor_with_same_amem(self, amem: AlphaMemory): + if self.amem == amem: + return self + return self.parent.find_nearest_ancestor_with_same_amem(amem) + + @property + def right_unlinked(self) -> bool: + return len(self.items) == 0 + + def left_activation(self, token: ReteToken, wme: WME, binding: Dict[V, Any]): + if not self.items: + self.relink_to_alpha_memory() + + new_token = ReteToken(parent=token, wme=wme, node=self, binding=binding) + self.items.append(new_token) + + for wme in self.amem.items: + if self.perform_join_test(new_token, wme): + jr = NegativeJoinResult(new_token, wme) + new_token.join_results.append(jr) + wme.negative_join_results.append(jr) + + if not new_token.join_results: + for child in self.children: + child.left_activation(new_token, None, binding) + + def right_activation(self, wme: WME): + for token in self.items: + if self.perform_join_test(token, wme): + if not token.join_results: + # TODO: TEST THIS - Chris ?? KSI - 2021-02-05 ??? + token.delete_descendents_of_token() + # t.delete_token_and_descendents() + jr = NegativeJoinResult(token, wme) + token.join_results.append(jr) + wme.negative_join_results.append(jr) diff --git a/src/sheerkarete/network.py b/src/sheerkarete/network.py new file mode 100644 index 0000000..6b47a26 --- /dev/null +++ b/src/sheerkarete/network.py @@ -0,0 +1,479 @@ +from __future__ import annotations + +from itertools import product +from typing import TYPE_CHECKING, Generator, Union + +from core.concept import Concept +from core.global_symbols import NotInit +from core.rule import Rule +from core.utils import as_bag +from sheerkarete.alpha import AlphaMemory +from sheerkarete.beta import ReteNode, BetaMemory +from sheerkarete.bind_node import BindNode +from sheerkarete.common import WME, Match, V +from sheerkarete.conditions import Condition, NegatedCondition, NegatedConjunctiveConditions, FilterCondition, \ + BindCondition +from sheerkarete.filter_node import FilterNode +from sheerkarete.join_node import JoinNode +from sheerkarete.ncc_node import NccNode, NccPartnerNode +from sheerkarete.negative_node import NegativeNode +from sheerkarete.pnode import PNode + +if TYPE_CHECKING: # pragma: no cover + from typing import Dict + from typing import Tuple + from typing import List + from typing import Set + from typing import Hashable + +FACT_ID = "##fact_id##" + + +class ReteNetwork: + """ + A Rete Network to store all the facts and productions to compute matches. + """ + + def __init__(self): + self.alpha_hash: Dict[Tuple[Hashable, Hashable, Hashable], List[AlphaMemory]] = {} + self.beta_root = ReteNode() + self.pnodes: List[PNode] = [] # list of all production nodes + self.rules: Set[Rule] = set() # set of all know rules + self.working_memory: Set[WME] = set() + + self.fact_counter: int = 0 + self.facts: Dict[str, object] = {} + + self.attributes_by_id = {} # keep track of requested conditions attributes, for a given id + self.default_attributes = set() # keep track of requested attributes, when the id is not given + + @property + def matches(self) -> Generator[Match, None, None]: + for pnode in self.pnodes: + for t in pnode.activations: + yield Match(pnode, t) + + def build_or_share_alpha_memory(self, condition): + """ + :type condition: Condition + :rtype: AlphaMemory + """ + + key = condition.get_key() + + # return existing alpha memory if it exists + if key in self.alpha_hash: + for amem in self.alpha_hash[key]: + if amem.condition == condition: + return amem + + # or create a new one + amem = AlphaMemory(key, condition) + self.alpha_hash.setdefault(key, []).append(amem) + + # fire already created WME if needed + for w in self.working_memory: + if condition.test(w): + amem.activation(w) + + return amem + + def build_or_share_beta_memory(self, parent): + """ + Create or reuse a BetaMemory + """ + # search for an existing one + for child in parent.children: + # if isinstance(child, BetaMemory): # Don't include subclasses + if type(child) == BetaMemory: + return child + + node = BetaMemory(parent=parent) + parent.children.append(node) + self.update_new_node_with_matches_from_above(node) + return node + + def build_or_share_join_node(self, parent: BetaMemory, amem: AlphaMemory, condition: Condition) -> JoinNode: + """ + Creates or reuse a JoinNode + :param parent: parent beta memory + :param amem: parent alpha memory + :param condition: condition for the join + :returns: + """ + + # search for already created join node + for child in parent.all_children: + if type(child) == JoinNode and child.amem == amem and child.condition == condition: + return child + + node = JoinNode(children=[], parent=parent, amem=amem, condition=condition) + + parent.children.append(node) + parent.all_children.append(node) + + amem.successors.append(node) + amem.reference_count += 1 + + node.update_nearest_ancestor_with_same_amem() + + # little optimisation. No need to bind if there is no wme in parent + if not parent.items: + amem.successors.remove(node) + elif not amem.items: + parent.children.remove(node) + + return node + + def build_or_share_negative_node(self, + parent: JoinNode, + amem: AlphaMemory, + condition: NegatedCondition) -> NegativeNode: + # search for already created join node + for child in parent.children: + if isinstance(child, NegativeNode) and child.amem == amem and child.condition == condition: + return child + + node = NegativeNode(parent=parent, amem=amem, condition=condition) + parent.children.append(node) + + amem.successors.append(node) + amem.reference_count += 1 + + node.update_nearest_ancestor_with_same_amem() + self.update_new_node_with_matches_from_above(node) + + # little optimisation. No need to bind if there is no wme in parent + if not node.items: + amem.successors.remove(node) + + return node + + def build_or_share_ncc_nodes(self, + parent: JoinNode, + ncc: NegatedConjunctiveConditions, + earlier_conds: List[Condition]) -> NccNode: + + # search for already created join node + bottom_of_subnetwork = self.build_or_share_network_for_conditions(parent, ncc, earlier_conds) + for child in parent.children: + if isinstance(child, NccNode) and child.partner.parent == bottom_of_subnetwork: + return child + + ncc_partner = NccPartnerNode(parent=bottom_of_subnetwork) + ncc_node = NccNode(partner=ncc_partner, children=[], parent=parent) + ncc_partner.ncc_node = ncc_node + parent.children.insert(0, ncc_node) + bottom_of_subnetwork.children.append(ncc_partner) + ncc_partner.number_of_conditions = ncc.number_of_conditions + self.update_new_node_with_matches_from_above(ncc_node) + self.update_new_node_with_matches_from_above(ncc_partner) + return ncc_node + + def build_or_share_filter_node(self, + parent: ReteNode, + f: FilterCondition) -> FilterNode: + # search for already created join node + for child in parent.children: + if isinstance(child, FilterNode) and child.func == f.func: + return child + + node = FilterNode([], parent, f.func, self) + parent.children.append(node) + return node + + def build_or_share_bind_node(self, + parent: ReteNode, + b: BindCondition) -> BindNode: + # search for already created join node + for child in parent.children: + if isinstance(child, BindNode) and child.func == b.func and child.bind == b.to: + return child + + node = BindNode([], parent, b.func, b.to, self) + parent.children.append(node) + + return node + + def build_or_share_p_node(self, parent: JoinNode, rule: Rule) -> Union[PNode, None]: + """ + Create or reuse a production node + :param parent: parent join node + :param rule: rule that will be fired on activation + :return: returns None if the PNode already exists + """ + for child in parent.children: + if isinstance(child, PNode): + child.rules.append(rule) + rule.rete_p_nodes.append(child) + return None + + node = PNode(rule=rule, parent=parent) + parent.children.append(node) + self.update_new_node_with_matches_from_above(node) + rule.rete_p_nodes.append(node) + return node + + def build_or_share_network_for_conditions(self, parent, conditions, earlier_conditions) -> ReteNode: + current_node = parent + conds_higher_up = earlier_conditions + + # Explanation on vars_ids_mappings + # conditions = [Condition(V("x"), "__name__", "fact_name"), + # Condition(V("x"), "attr1", "value1"), + # Condition(V("x"), "attr1", "value1")] + # V(x) actually refers to a object named 'fact_name' + # self.conditions_attributes_by_id must be updated accordingly + vars_ids_mappings = {} + + for cond in conditions: + # update requested attributes for a fact + if isinstance(cond, Condition): + # Manage list of requested attributes + if isinstance(cond.identifier, V) and cond.attribute == "__name__": + vars_ids_mappings[cond.identifier] = cond.value + + identifier = vars_ids_mappings[cond.identifier] if cond.identifier in vars_ids_mappings else \ + cond.identifier if not isinstance(cond.identifier, V) else \ + None + if identifier: + attr = "*" if isinstance(cond.attribute, V) else cond.attribute + self.attributes_by_id.setdefault(identifier, []).append(attr) + elif not isinstance(cond.attribute, V): + self.default_attributes.add(cond.attribute) + + # create the alpha memory (if needed), beta memory and join node + if isinstance(cond, Condition) and not isinstance(cond, NegatedCondition): + am = self.build_or_share_alpha_memory(cond) + current_node = self.build_or_share_beta_memory(current_node) + current_node = self.build_or_share_join_node(current_node, am, cond) + + elif isinstance(cond, NegatedCondition): + am = self.build_or_share_alpha_memory(cond) + current_node = self.build_or_share_negative_node(current_node, am, cond) + + elif isinstance(cond, NegatedConjunctiveConditions): + current_node = self.build_or_share_ncc_nodes(current_node, cond, conds_higher_up) + + elif isinstance(cond, FilterCondition): + current_node = self.build_or_share_filter_node(current_node, cond) + + elif isinstance(cond, BindCondition): + current_node = self.build_or_share_bind_node(current_node, cond) + + conds_higher_up.append(cond) + + return current_node + + def get_rete_conditions(self, rule): + """ + Gets the conditions from a rule + It's in fact the list of disjunctions + Not sure yet which component will hold this functionality + """ + if hasattr(rule, "get_rete_disjunctions"): + return rule.get_rete_disjunctions() + + raise NotImplementedError("") + + def add_rule(self, rule: Rule): + + if rule.id is None: + raise ValueError("Rule has no id, cannot add") + + if not rule.metadata.is_enabled or not rule.metadata.is_compiled: + return + + if rule.rete_net: + raise ValueError("Rule is already added") + + rule.rete_net = self + self.rules.add(rule) + + for full_condition in self.get_rete_conditions(rule): + conditions = full_condition.conditions + current_node = self.build_or_share_network_for_conditions(self.beta_root, conditions, []) + p_node = self.build_or_share_p_node(current_node, rule) + if p_node is not None: + self.pnodes.append(p_node) + + def remove_rule(self, rule: Rule): + """ + Removes a pnode from the network + """ + if rule.rete_net is None: + return + + # Remove production + self.rules.remove(rule) + + for pnode in rule.rete_p_nodes: + pnode.rules.remove(rule) + if len(pnode.rules) == 0: + self.delete_node_and_any_unused_ancestors(pnode) + self.pnodes.remove(pnode) + + rule.p_nodes = [] + + def add_wme(self, wme: WME) -> None: + if wme in self.working_memory: + return + + keys = product([wme.identifier, '*'], + [wme.attribute, '*'], + [wme.value, '*']) + + for key in keys: + if key in self.alpha_hash: + for amem in reversed(self.alpha_hash[key]): + amem.activation(wme) + + self.working_memory.add(wme) + + def remove_wme(self, wme: WME) -> None: + for stored_wme in self.working_memory: + if wme == stored_wme: + wme = stored_wme + break + + for am in wme.amems: + am.items.remove(wme) + if not am.items: + for node in am.successors: + if isinstance(node, JoinNode) and not isinstance(node, NegativeNode): + node.parent.children.remove(node) + + while wme.tokens: + t = wme.tokens[0] + t.delete_token_and_descendents() + + for jr in wme.negative_join_results: + jr.owner.join_results.remove(jr) + if not jr.owner.join_results: + if jr.owner.node and jr.owner.node.children is not None: + for child in jr.owner.node.children: + child.left_activation(jr.owner, None, jr.owner.binding) + + self.working_memory.remove(wme) + + def remove_wme_by_fact_id(self, identifier: str) -> None: + to_remove = [wme for wme in self.working_memory if wme.identifier == + identifier or wme.identifier.startswith(identifier + ".")] + for wme in to_remove: + self.remove_wme(wme) + + def add_obj(self, name, obj, use_bag=False, root=True): + """ + Adds a new object to the working memory + """ + + def inner_add_vme(ident, attr, val): + if val is NotInit: + pass + elif attr != "self" and isinstance(val, Concept): + new_name = f"{ident}.{attr}" + self.add_wme(WME(ident, attr, new_name)) + self.add_obj(new_name, val, use_bag=True, root=False) + else: + self.add_wme(WME(ident, attr, val)) + + if root: + if hasattr(obj, FACT_ID): + raise ValueError("Object already has an id, cannot add") + + fact_id = f"f-{self.fact_counter:05}" + setattr(obj, FACT_ID, fact_id) + self.facts[fact_id] = obj + self.fact_counter += 1 + else: + fact_id = name + + requested_attributes = "*" if use_bag else \ + self.attributes_by_id[name] if name in self.attributes_by_id else \ + self.default_attributes + + for attribute in requested_attributes: + if attribute == "*": + bag = as_bag(obj) + for k, v in bag.items(): + inner_add_vme(fact_id, k, v) + elif attribute == "__name__": + self.add_wme(WME(fact_id, "__name__", name)) + else: + try: + value = getattr(obj, attribute) + inner_add_vme(fact_id, attribute, value) + except AttributeError: + pass + + def remove_obj(self, obj): + if not hasattr(obj, FACT_ID) or (fact_id := getattr(obj, FACT_ID)) not in self.facts: + raise ValueError("Fact has no id or does not exist in network.") + + self.remove_wme_by_fact_id(fact_id) + del self.facts[fact_id] + delattr(obj, FACT_ID) + + def update_new_node_with_matches_from_above(self, new_node: ReteNode) -> None: + parent = new_node.parent + if parent == self.beta_root: + new_node.left_activation(None, None, {}) + elif isinstance(parent, BetaMemory) and not isinstance(parent, (NccNode, NegativeNode)): + for tok in parent.items: + new_node.left_activation(token=tok) + elif isinstance(parent, JoinNode) and not isinstance(parent, NegativeNode): + saved_list_of_children = parent.children + parent.children = [new_node] + for item in parent.amem.items: + parent.right_activation(item) + parent.children = saved_list_of_children + elif isinstance(parent, NegativeNode): + for token in parent.items: + if not token.join_results: + new_node.left_activation(token, None, token.binding) + elif isinstance(parent, NccNode): + for token in parent.items: + if not token.ncc_results: + new_node.left_activation(token, None, token.binding) + elif isinstance(parent, (BindNode, FilterNode)): + saved_list_of_children = parent.children + parent.children = [new_node] + self.update_new_node_with_matches_from_above(parent) + parent.children = saved_list_of_children + + def delete_alpha_memory(self, amem: AlphaMemory): + del self.alpha_hash[amem.key] + + def delete_node_and_any_unused_ancestors(self, node: ReteNode): + if isinstance(node, NccNode): + self.delete_node_and_any_unused_ancestors(node.partner) + + if isinstance(node, BetaMemory): + for item in node.items: + item.delete_token_and_descendents() + + if isinstance(node, NccPartnerNode): + for item in node.new_result_buffer: + item.delete_token_and_descendents() + + if isinstance(node, JoinNode) and not isinstance(node, NegativeNode): + if not node.right_unlinked: + node.amem.successors.remove(node) + + node.amem.reference_count -= 1 + + if node.amem.reference_count == 0: + self.delete_alpha_memory(node.amem) + + if not node.left_unlinked: + node.parent.children.remove(node) + + node.parent.all_children.remove(node) + + if not node.parent.all_children: + self.delete_node_and_any_unused_ancestors(node.parent) + + elif node.parent: + node.parent.children.remove(node) + if not node.parent.children: + self.delete_node_and_any_unused_ancestors(node.parent) diff --git a/src/sheerkarete/pnode.py b/src/sheerkarete/pnode.py new file mode 100644 index 0000000..21bf104 --- /dev/null +++ b/src/sheerkarete/pnode.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from core.rule import Rule +from sheerkarete.beta import BetaMemory +from sheerkarete.common import ReteToken + +if TYPE_CHECKING: # pragma: no cover + from typing import List + from typing import Dict + from typing import Any + from sheerkarete.common import WME + from sheerkarete.common import V + + +class PNode(BetaMemory): + """ + A beta network node that stores the matches for productions. + """ + + def __init__(self, rule: Rule, **kwargs): + super(PNode, self).__init__(**kwargs) + self.rules = [rule] + self.new: List[ReteToken] = [] + + def left_activation(self, token: ReteToken, wme: WME, binding: Dict[V, Any]): + new_token = ReteToken(token, wme, node=self, binding=binding) + self.items.append(new_token) + self.new.append(new_token) + + def pop_new_token(self): + if self.new: + return self.new.pop() + + def new_activations(self): + while self.new: + t = self.new.pop() + yield t + + @property + def activations(self): + for t in self.items: + yield t diff --git a/tests/BaseTest.py b/tests/BaseTest.py index 254b729..5cfe145 100644 --- a/tests/BaseTest.py +++ b/tests/BaseTest.py @@ -3,7 +3,7 @@ from dataclasses import dataclass, field from core.builtin_concepts import ReturnValueConcept, ParserResultConcept, BuiltinConcepts from core.concept import Concept, DEFINITION_TYPE_BNF, DEFINITION_TYPE_DEF, freeze_concept_attrs -from core.rule import Rule, ACTION_TYPE_PRINT +from core.rule import Rule, ACTION_TYPE_PRINT, ACTION_TYPE_EXEC from core.sheerka.ExecutionContext import ExecutionContext from core.sheerka.Sheerka import Sheerka from core.sheerka.services.SheerkaRuleManager import SheerkaRuleManager @@ -59,16 +59,22 @@ class InitTestHelper: return self - def with_rules(self, *rules, **kwargs): + def with_format_rules(self, *rules, **kwargs): + return self.with_rules(ACTION_TYPE_PRINT, *rules, **kwargs) + + def with_exec_rules(self, *rules, **kwargs): + return self.with_rules(ACTION_TYPE_EXEC, *rules, **kwargs) + + def with_rules(self, action_type, *rules, **kwargs): create_new = kwargs.get("create_new", True) compile_rule = kwargs.get("compile_rule", True) for rule_template in rules: if isinstance(rule_template, tuple): if len(rule_template) == 3: - rule = Rule(ACTION_TYPE_PRINT, rule_template[0], rule_template[1], rule_template[2]) + rule = Rule(action_type, rule_template[0], rule_template[1], rule_template[2]) else: - rule = Rule(ACTION_TYPE_PRINT, None, rule_template[0], rule_template[1]) + rule = Rule(action_type, None, rule_template[0], rule_template[1]) else: rule = rule_template @@ -150,7 +156,7 @@ class BaseTest: return self.init_test(**init_test_args).with_concepts(*concepts, **with_concepts_args).unpack() def init_format_rules(self, *rules, **kwargs): - return self.init_test(**kwargs).with_rules(*rules, **kwargs).unpack() + return self.init_test(**kwargs).with_format_rules(*rules, **kwargs).unpack() @staticmethod def get_concept_instance(sheerka, concept, **kwargs): diff --git a/tests/TestUsingFileBasedSheerka.py b/tests/TestUsingFileBasedSheerka.py index e7c514b..673f81e 100644 --- a/tests/TestUsingFileBasedSheerka.py +++ b/tests/TestUsingFileBasedSheerka.py @@ -1,4 +1,5 @@ from conftest import SHEERKA_TEST_FOLDER +from core.global_symbols import EVENT_ONTOLOGY_CREATED from core.sheerka.Sheerka import Sheerka from core.sheerka.SheerkaOntologyManager import SheerkaOntologyManager @@ -10,19 +11,28 @@ class TestUsingFileBasedSheerka(BaseTest): context = None root_ontology_name = SheerkaOntologyManager.ROOT_ONTOLOGY_NAME + ontologies_created = set() + def teardown_method(self, method): # to do after the test if TestUsingFileBasedSheerka.sheerka: while TestUsingFileBasedSheerka.sheerka.om.current_ontology().name != self.root_ontology_name: - ontology = TestUsingFileBasedSheerka.sheerka.pop_ontology().body.body + ontology = TestUsingFileBasedSheerka.sheerka.pop_ontology(self.context).body.body ontology.cache_manager.sdp.test_only_destroy_refs() + # remove other created ontologies + for ontology in TestUsingFileBasedSheerka.ontologies_created: + TestUsingFileBasedSheerka.sheerka.om.current_sdp().test_only_destroy_refs(ontology) + TestUsingFileBasedSheerka.ontologies_created.clear() + @staticmethod def new_sheerka_instance(cache_only): sheerka = Sheerka(cache_only=cache_only) sheerka.initialize(SHEERKA_TEST_FOLDER, save_execution_context=False, - enable_process_return_values=False) + enable_process_return_values=False, + enable_process_rules=False) + sheerka.subscribe(EVENT_ONTOLOGY_CREATED, lambda c, o: TestUsingFileBasedSheerka.ontologies_created.add(o)) return sheerka def get_sheerka(self, **kwargs) -> Sheerka: diff --git a/tests/TestUsingMemoryBasedSheerka.py b/tests/TestUsingMemoryBasedSheerka.py index f000f69..0e4b998 100644 --- a/tests/TestUsingMemoryBasedSheerka.py +++ b/tests/TestUsingMemoryBasedSheerka.py @@ -13,12 +13,15 @@ class TestUsingMemoryBasedSheerka(BaseTest): # to do after the test if TestUsingMemoryBasedSheerka.sheerka: while TestUsingMemoryBasedSheerka.sheerka.om.current_ontology().name != self.root_ontology_name: - TestUsingMemoryBasedSheerka.sheerka.pop_ontology() + TestUsingMemoryBasedSheerka.sheerka.pop_ontology(self.context) @staticmethod def new_sheerka_instance(cache_only): sheerka = Sheerka(cache_only=cache_only) - sheerka.initialize("mem://", save_execution_context=False, enable_process_return_values=False) + sheerka.initialize("mem://", + save_execution_context=False, + enable_process_return_values=False, + enable_process_rules=False) return sheerka def get_sheerka(self, **kwargs) -> Sheerka: diff --git a/tests/core/test_SheerkaComparisonManager.py b/tests/core/test_SheerkaComparisonManager.py index e39059e..5da6fa2 100644 --- a/tests/core/test_SheerkaComparisonManager.py +++ b/tests/core/test_SheerkaComparisonManager.py @@ -65,9 +65,9 @@ class TestSheerkaGreaterThanManager(TestUsingMemoryBasedSheerka): assert in_db == [ComparisonObj(context.event.get_digest(), "prop_name", two.str_id, one.str_id, ">", "#")] def test_i_can_add_is_greater_than_for_rules(self): - sheerka, context, r1, r2 = self.init_test(cache_only=False).with_rules(("True", "true"), - ("False", "false"), - compile_rule=False).unpack() + sheerka, context, r1, r2 = self.init_test(cache_only=False).with_format_rules(("True", "true"), + ("False", "false"), + compile_rule=False).unpack() service = sheerka.services[SheerkaComparisonManager.NAME] @@ -106,10 +106,10 @@ class TestSheerkaGreaterThanManager(TestUsingMemoryBasedSheerka): assert in_db == [ComparisonObj(context.event.get_digest(), "prop_name", one.str_id, two.str_id, "<", "#")] def test_i_can_add_is_less_than_for_rules(self): - sheerka, context, r1, r2 = self.init_test(cache_only=False).with_rules(("True", "true"), - ("False", "false"), - compile_rule=False, - create_new=True).unpack() + sheerka, context, r1, r2 = self.init_test(cache_only=False).with_format_rules(("True", "true"), + ("False", "false"), + compile_rule=False, + create_new=True).unpack() service = sheerka.services[SheerkaComparisonManager.NAME] @@ -213,7 +213,7 @@ class TestSheerkaGreaterThanManager(TestUsingMemoryBasedSheerka): assert sheerka.om.get(SheerkaComparisonManager.RESOLVED_COMPARISON_ENTRY, "prop_name|#") == expected_weights2 # I can retrieve the previous values - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.om.get(SheerkaComparisonManager.COMPARISON_ENTRY, "prop_name|#") == expected_in_cache assert sheerka.om.get(SheerkaComparisonManager.RESOLVED_COMPARISON_ENTRY, "prop_name|#") == expected_weights @@ -544,9 +544,9 @@ class TestSheerkaGreaterThanManager(TestUsingMemoryBasedSheerka): assert event_received def test_an_event_is_fired_when_modifying_rule_precedence(self): - sheerka, context, r1, r2 = self.init_test(cache_only=False).with_rules(("True", "true"), - ("False", "false"), - compile_rule=False).unpack() + sheerka, context, r1, r2 = self.init_test(cache_only=False).with_format_rules(("True", "true"), + ("False", "false"), + compile_rule=False).unpack() foo = Concept("foo") event_received = False sheerka.om.clear(SheerkaComparisonManager.COMPARISON_ENTRY) diff --git a/tests/core/test_SheerkaConceptManager.py b/tests/core/test_SheerkaConceptManager.py index e313b1e..3992a7c 100644 --- a/tests/core/test_SheerkaConceptManager.py +++ b/tests/core/test_SheerkaConceptManager.py @@ -563,7 +563,7 @@ class TestSheerkaConceptManager(TestUsingMemoryBasedSheerka): assert updated.get_metadata().variables == [("a", None), ("c", "value")] assert updated.get_metadata().props == {BuiltinConcepts.ISA: {"bar"}} - sheerka.pop_ontology() + sheerka.pop_ontology(context) def test_i_cannot_modify_without_any_modification(self): sheerka, context, foo = self.init_concepts("foo") @@ -726,7 +726,7 @@ class TestSheerkaConceptManager(TestUsingMemoryBasedSheerka): assert not res.status assert sheerka.isinstance(res.body, BuiltinConcepts.CONCEPT_ALREADY_DEFINED) - sheerka.pop_ontology() + sheerka.pop_ontology(context) # But I can if I remove the layer res = sheerka.create_new_concept(context, Concept("bar")) assert res.status diff --git a/tests/core/test_SheerkaDebugManager.py b/tests/core/test_SheerkaDebugManager.py index 5465d41..002d4b4 100644 --- a/tests/core/test_SheerkaDebugManager.py +++ b/tests/core/test_SheerkaDebugManager.py @@ -1,12 +1,11 @@ import pytest + from core.builtin_concepts import BuiltinConcepts from core.concept import Concept from core.global_symbols import NotInit, NotFound from core.sheerka.ExecutionContext import ExecutionContext from core.sheerka.services.SheerkaDebugManager import SheerkaDebugManager, DebugItem, ConceptDebugObj from parsers.PythonParser import PythonNode -from sdp.sheerkaDataProvider import Event - from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka @@ -21,220 +20,12 @@ class DummyObj: class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): - def test_i_can_activate_debug(self): - sheerka, context = self.init_concepts() + return_value_id = 0 - sheerka.set_debug(context, True) - assert sheerka.debug_activated() - - sheerka.set_debug(context, False) - assert not sheerka.debug_activated() - - def test_when_debug_mode_is_activated_context_are_in_debug_mode(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - - sheerka.set_debug(root_context, True) - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - sub_context = context.push(BuiltinConcepts.NOP, None) # sub_sub_context.parent = 1 - sub_context2 = context.push(BuiltinConcepts.NOP, None) # sub_sub_context2.parent = 1 - sub_sub_context = sub_context.push(BuiltinConcepts.NOP, None) # is a child - - assert context.id == 1 - assert context.debug_enabled - assert sub_context.debug_enabled - assert sub_context2.debug_enabled - assert sub_sub_context.debug_enabled - assert not root_context.debug_enabled - - def test_i_can_activate_debug_for_new_context(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1) - - context = ExecutionContext("test", Event(), sheerka, BuiltinConcepts.TESTING, None) # context.id = 1 - assert context.debug_enabled - - def test_i_can_activate_debug_for_new_context_2(self): - """ - This time children is also requested - :return: - """ - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1, children=True) - - context = ExecutionContext("test", Event(), sheerka, BuiltinConcepts.TESTING, None) # context.id = 1 - assert context.debug_enabled - - def test_i_can_activate_debug_for_a_context_using_push(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1) - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - sub_context = context.push(BuiltinConcepts.NOP, None) # sub_sub_context.parent = 1 - - assert context.id == 1 - assert 1 in service.context_cache - assert context.debug_enabled - assert not sub_context.debug_enabled - assert not root_context.debug_enabled - - def test_global_debug_must_be_activated_to_activate_context_debug(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, False) - sheerka.activate_debug_for(root_context, 1) - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - - assert context.id == 1 - assert 1 in service.context_cache - assert not context.debug_enabled - - def test_i_can_activate_debug_for_sub_children(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1, children=True) - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - sub_context = context.push(BuiltinConcepts.NOP, None) # sub_sub_context.parent = 1 - sub_context2 = context.push(BuiltinConcepts.NOP, None) # sub_sub_context2.parent = 1 - sub_sub_context = sub_context.push(BuiltinConcepts.NOP, None) # is a child - - assert context.id == 1 - assert 1 in service.context_cache - assert "1+" in service.context_cache - assert context.debug_enabled - assert sub_context.debug_enabled - assert sub_context2.debug_enabled - assert sub_sub_context.debug_enabled - assert not root_context.debug_enabled - - def test_i_can_deactivate_debug_for_a_context(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1) - sheerka.deactivate_debug_for(root_context, 1) - - context = root_context.push(BuiltinConcepts.NOP, None) - - assert context.id == 1 - assert 1 not in service.context_cache - assert not context.debug_enabled - - def test_i_can_deactivate_context_but_not_children(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1, True) - sheerka.deactivate_debug_for(root_context, 1) - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - sub_context = context.push(BuiltinConcepts.NOP, None) # sub_sub_context.parent = 1 - sub_context2 = context.push(BuiltinConcepts.NOP, None) # sub_sub_context2.parent = 1 - sub_sub_context = sub_context.push(BuiltinConcepts.NOP, None) # is a child - - assert context.id == 1 - assert 1 not in service.context_cache - assert "1+" in service.context_cache - assert not context.debug_enabled - assert sub_context.debug_enabled - assert sub_context2.debug_enabled - assert sub_sub_context.debug_enabled - - def test_i_can_deactivate_context_and_children(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, 1) - sheerka.deactivate_debug_for(root_context, 1, children=True) - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - sub_context = context.push(BuiltinConcepts.NOP, None) # sub_sub_context.parent = 1 - sub_context2 = context.push(BuiltinConcepts.NOP, None) # sub_sub_context2.parent = 1 - sub_sub_context = sub_context.push(BuiltinConcepts.NOP, None) # is a child - - assert 1 not in service.context_cache - assert "1+" not in service.context_cache - assert not context.debug_enabled - assert not sub_context.debug_enabled - assert not sub_context2.debug_enabled - assert not sub_sub_context.debug_enabled - - def test_i_can_activate_debug_for_a_variable(self): - sheerka, context = self.init_concepts() - service = sheerka.services[SheerkaDebugManager.NAME] - sheerka.set_debug(context) - - sheerka.activate_debug_for(context, "Out") - assert "Out" in service.variable_cache - assert sheerka.debug_activated_for("Out") - - sheerka.deactivate_debug_for(context, "Out") - assert "Out" not in service.variable_cache - assert not sheerka.debug_activated_for("Out") - - def test_i_can_activate_debug_for_sub_children_using_the_simplified_form(self): - sheerka = self.get_sheerka() - ExecutionContext.ids.clear() - root_context = self.get_context(sheerka) - service = sheerka.services[SheerkaDebugManager.NAME] - - sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, True) - sheerka.activate_debug_for(root_context, "1+") - - context = root_context.push(BuiltinConcepts.NOP, None) # sub_context.id = 1 - sub_context = context.push(BuiltinConcepts.NOP, None) # sub_sub_context.parent = 1 - sub_context2 = context.push(BuiltinConcepts.NOP, None) # sub_sub_context2.parent = 1 - sub_sub_context = sub_context.push(BuiltinConcepts.NOP, None) # is a child - - assert context.id == 1 - assert 1 in service.context_cache - assert "1+" in service.context_cache - assert context.debug_enabled - assert sub_context.debug_enabled - assert sub_context2.debug_enabled - assert sub_sub_context.debug_enabled - assert not root_context.debug_enabled + @classmethod + def setup(cls): + sheerka = cls().get_sheerka() + cls.return_value_id = sheerka.get_by_key("__RETURN_VALUE").id @pytest.mark.parametrize("item_type", [ "vars", "rules", "concepts" @@ -631,9 +422,6 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): root_context = self.get_context(sheerka) sheerka.set_debug(root_context, True) - sheerka.set_explicit(root_context, False) - sheerka.activate_debug_for(root_context, 1, children=True) - sheerka.activate_debug_for(root_context, "SomeVar") sheerka.debug_var(root_context, "service_name.*.var") sheerka.debug_rule(root_context, 1) sheerka.debug_concept(root_context, 1001) @@ -642,9 +430,6 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): another_service.initialize_deferred(root_context, True) assert another_service.activated - assert not another_service.explicit - assert another_service.context_cache == {1, "1+"} - assert another_service.variable_cache == {"SomeVar"} assert another_service.debug_vars_settings == [ DebugItem('var', 'service_name', None, None, False, None, False, True)] assert another_service.debug_rules_settings == [ @@ -688,7 +473,7 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): python_node = PythonNode("one + 1").init_ast() res = sheerka.inspect(context, python_node) - assert set(res.body.keys()) == {"#type#", 'ast_', 'ast_str', 'compiled', 'objects', 'source'} + assert set(res.body.keys()) == {"#type#", 'ast_', 'ast_str', 'compiled', 'objects', 'source', 'original_source'} def test_i_can_inspect_object_specified_attributes(self): sheerka, context = self.init_concepts() @@ -872,7 +657,7 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): assert res.body == { 'body': concept_debug_obj, '#type#': 'ReturnValueConcept', - 'id': '43', + 'id': f'{self.return_value_id}', 'key': '__RETURN_VALUE', 'name': '__RETURN_VALUE', 'parents': [concept_debug_obj], @@ -920,21 +705,18 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): sheerka.push_ontology(context, "new ontology") service.set_debug(context) - service.set_explicit(context) service.debug_var(context, "var_service.var_method.var_name", "1+", 1) service.debug_rule(context, "rule_service.rule_method.rule_name", "2+", 2) service.debug_concept(context, "concept_service.concept_method.concept_name", "3+", 3) # sanity check assert service.activated - assert service.explicit assert service.debug_vars_settings != [] assert service.debug_rules_settings != [] assert service.debug_concepts_settings != [] - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert not service.activated - assert not service.explicit assert service.context_cache == set() assert service.variable_cache == set() assert service.debug_vars_settings == [] @@ -946,7 +728,6 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): service = sheerka.services[SheerkaDebugManager.NAME] service.set_debug(context) - service.set_explicit(context) service.debug_var(context, "v_service.v_method.v_name", "1+", 1) service.debug_rule(context, "r_service.r_method.r_name", "2+", 2) service.debug_concept(context, "c_serv.c_method.c_name", "3+", 3) @@ -955,7 +736,6 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): # modify the state service.set_debug(context, False) - service.set_explicit(context, False) service.debug_var(context, "var_service2.var_method2.var_name2", "11+", 11) service.debug_rule(context, "rule_service2.rule_method2.rule_name2", "22+", 22) service.debug_concept(context, "concept_service2.concept_method2.concept_name2", "33+", 33) @@ -967,10 +747,8 @@ class TestSheerkaDebugManager(TestUsingMemoryBasedSheerka): assert len(service.debug_rules_settings) == 2 assert len(service.debug_concepts_settings) == 2 - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert service.activated - assert service.explicit assert service.debug_vars_settings == [DebugItem("v_name", "v_service", "v_method", 1, True, 1, False, True)] assert service.debug_rules_settings == [DebugItem("r_name", "r_service", "r_method", 2, True, 2, False, True)] assert service.debug_concepts_settings == [DebugItem("c_name", "c_serv", "c_method", 3, True, 3, False, True)] - diff --git a/tests/core/test_SheerkaEvaluateRules.py b/tests/core/test_SheerkaEvaluateRules.py index 2864baf..21ca0b6 100644 --- a/tests/core/test_SheerkaEvaluateRules.py +++ b/tests/core/test_SheerkaEvaluateRules.py @@ -1,15 +1,22 @@ import operator -from core.concept import Concept -from core.rule import Rule -from core.sheerka.services.SheerkaEvaluateRules import SheerkaEvaluateRules, LOW_PRIORITY_RULES, DISABLED_RULES +import pytest +from core.builtin_concepts_ids import BuiltinConcepts +from core.concept import Concept, DEFINITION_TYPE_DEF +from core.rule import Rule, ACTION_TYPE_EXEC +from core.sheerka.Sheerka import RECOGNIZED_BY_ID, RECOGNIZED_BY_NAME +from core.sheerka.services.SheerkaEvaluateRules import SheerkaEvaluateRules, LOW_PRIORITY_RULES, DISABLED_RULES +from core.sheerka.services.SheerkaExecute import ParserInput +from core.sheerka.services.SheerkaRuleManager import RuleCompiledPredicate, SheerkaRuleManager +from evaluators.PythonEvaluator import PythonEvaluator, Expando +from parsers.PythonParser import PythonParser from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka class TestSheerkaEvaluateRules(TestUsingMemoryBasedSheerka): def test_i_can_evaluate_python_rules(self): - sheerka, context, r1, r2, r3, r4, r5, r6, r7, r8, r9 = self.init_test().with_rules( + sheerka, context, r1, r2, r3, r4, r5, r6, r7, r8, r9 = self.init_test().with_format_rules( Rule(predicate="a == 1", action="", priority=1), # r1 Rule(predicate="a == 2", action="", priority=1), # r2 Rule(predicate="a == 3", action="", priority=0), # r3 @@ -32,9 +39,10 @@ class TestSheerkaEvaluateRules(TestUsingMemoryBasedSheerka): DISABLED_RULES: [r7] } - def test_i_can_evaluate_concept_rules(self): + def test_i_can_evaluate_question_concept_rules(self): sheerka, context, concept, r1, r2, r3, r4, r5, r6, r7, r8, r9 = self.init_test().with_concepts( - Concept("x equals y", body="x == y").def_var("x").def_var("y"), create_new=True).with_rules( + Concept("x equals y", body="x == y", pre="is_question()").def_var("x").def_var("y"), + create_new=True).with_format_rules( Rule(predicate="a equals 1", action="", priority=1), # r1 Rule(predicate="a equals 2", action="", priority=1), # r2 Rule(predicate="a equals 3", action="", priority=0), # r3 @@ -58,6 +66,159 @@ class TestSheerkaEvaluateRules(TestUsingMemoryBasedSheerka): DISABLED_RULES: [r7] } + @pytest.mark.parametrize("predicates", [ + ("True", "False"), + ("False", "True"), + ]) + def test_i_can_eval_when_multiple_python_predicates(self, predicates): + """ + In this test, the rule has multiple predicates to evaluate + The test make sure that is one predicate is successful, the rule must be evaluated + """ + sheerka, context, my_rule = self.init_test().with_format_rules(Rule("my rule"), + compile_rule=False, + create_new=False).unpack() + service = sheerka.services[SheerkaEvaluateRules.NAME] + + # create fake compiled predicates + parser = PythonParser() + my_rule.compiled_predicates = [ + RuleCompiledPredicate("my rule", None, PythonEvaluator.NAME, parser.parse(context, ParserInput(exp)), None) + for exp in predicates] + my_rule.metadata.is_compiled = True + my_rule.metadata.is_enabled = True + + res = service.evaluate_rules(context, [my_rule], {}, set()) + + assert res == { + True: [my_rule], + } + + def test_i_can_evaluate_rules_when_concepts_are_questions(self): + sheerka, context, isa, cat, crocodile, pet, r1, r2, r3 = self.init_test().with_concepts( + Concept("x is a y", body="isa(x,y)", pre="is_question()").def_var("x").def_var("y"), + "cat", + "crocodile", + "pet", + create_new=True).with_format_rules( + Rule(predicate="cat is a pet", action=""), + Rule(predicate="crocodile is a pet", action=""), + Rule(predicate="not crocodile is a pet", action=""), + ).unpack() + sheerka.set_isa(context, cat, pet) + service = sheerka.services[SheerkaEvaluateRules.NAME] + + res = service.evaluate_rules(context, [r1, r2, r3], {}, set()) + assert res == {True: [r1, r3], False: [r2]} + + @pytest.mark.parametrize("predicate", [ + "greetings", + "c:|1001:", + "hello 'kodjo'" + ]) + def test_i_can_evaluate_rules_when_concepts_are_not_questions(self, predicate): + """ + In this test, we evaluate rules that involves concepts that are not questions + """ + sheerka, context, greetings, rule = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + create_new=True).with_format_rules( + Rule(predicate=predicate, action="")).unpack() + + service = sheerka.services[SheerkaEvaluateRules.NAME] + ret = sheerka.ret("evaluator", True, sheerka.new(greetings, a="kodjo")) + + res = service.evaluate_rules(context, [rule], {"__ret": ret}, set()) + assert res == {True: [rule]} + + @pytest.mark.parametrize("recognized_by", [ + RECOGNIZED_BY_ID, + RECOGNIZED_BY_NAME, + None + ]) + def test_i_can_evaluate_concept_rules_when_variable_is_a_one_word_concept(self, recognized_by): + """ + In this test, we evaluate rules that involves concepts that are not questions + """ + sheerka, context, greetings, there, rule = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + Concept("there"), + create_new=True).with_format_rules( + Rule(predicate="hello there", action="")).unpack() + service = sheerka.services[SheerkaEvaluateRules.NAME] + + there_instance = sheerka.new_from_template(there, there.key) + if recognized_by: + there_instance.set_hint(BuiltinConcepts.RECOGNIZED_BY, recognized_by) + ret = sheerka.ret("evaluator", True, sheerka.new(greetings, a=there_instance)) + res = service.evaluate_rules(context, [rule], {"__ret": ret}, set()) + assert res == {True: [rule]} + + @pytest.mark.parametrize("recognized_by", [ + RECOGNIZED_BY_ID, + RECOGNIZED_BY_NAME, + None + ]) + def test_i_can_evaluate_concept_rules_when_variable_is_a_two_words_concept(self, recognized_by): + """ + In this test, we evaluate rules that involves concepts that are not questions + """ + sheerka, context, greetings, my_friend, rule = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + Concept("my friend"), + create_new=True).with_format_rules( + Rule(predicate="hello my friend", action="")).unpack() + service = sheerka.services[SheerkaEvaluateRules.NAME] + + my_friend_instance = sheerka.new_from_template(my_friend, my_friend.key) + if recognized_by: + my_friend_instance.set_hint(BuiltinConcepts.RECOGNIZED_BY, recognized_by) + ret = sheerka.ret("evaluator", True, sheerka.new(greetings, a=my_friend_instance)) + res = service.evaluate_rules(context, [rule], {"__ret": ret}, set()) + assert res == {True: [rule]} + + def test_i_can_evaluate_concept_rules_when_variable_is_an_expando(self): + sheerka, context, greetings, rule = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + create_new=True).with_format_rules( + Rule(predicate="hello sheerka", action="")).unpack() + service = sheerka.services[SheerkaEvaluateRules.NAME] + + ret = sheerka.ret("evaluator", True, sheerka.new(greetings, a=Expando("sheerka", {}))) + res = service.evaluate_rules(context, [rule], {"__ret": ret}, set()) + assert res == {True: [rule]} + + def test_i_can_evaluate_concept_rules_when_same_name(self): + sheerka, context, g1, g2, rule = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + Concept("greetings", definition="hi a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + create_new=True).with_format_rules( + Rule(predicate="greetings", action="")).unpack() + + service = sheerka.services[SheerkaEvaluateRules.NAME] + ret1 = sheerka.ret("evaluator", True, sheerka.new(g1, a="kodjo")) + res = service.evaluate_rules(context, [rule], {"__ret": ret1}, set()) + assert res == {True: [rule]} + + ret2 = sheerka.ret("evaluator", True, sheerka.new(g2, a="kodjo")) + res = service.evaluate_rules(context, [rule], {"__ret": ret2}, set()) + assert res == {True: [rule]} + + def test_i_can_evaluate_concept_rule_with_the_same_name_when_the_second_concept_is_declared_after(self): + sheerka, context, g1, rule, g2 = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + create_new=True).with_format_rules(Rule(predicate="greetings", action="")).with_concepts( + Concept("greetings", definition="hi a", definition_type=DEFINITION_TYPE_DEF).def_var("a")).unpack() + + service = sheerka.services[SheerkaEvaluateRules.NAME] + ret1 = sheerka.ret("evaluator", True, sheerka.new(g1, a="kodjo")) + res = service.evaluate_rules(context, [rule], {"__ret": ret1}, set()) + assert res == {True: [rule]} + + ret2 = sheerka.ret("evaluator", True, sheerka.new(g2, a="kodjo")) + res = service.evaluate_rules(context, [rule], {"__ret": ret2}, set()) + assert res == {True: [rule]} + def test_i_can_disable_rules_at_runtime(self): sheerka, context, r1, r2, = self.init_format_rules( Rule(predicate="a == 1", action="", priority=2), # r1 @@ -72,3 +233,15 @@ class TestSheerkaEvaluateRules(TestUsingMemoryBasedSheerka): True: [r2], DISABLED_RULES: [r1] } + + def test_rete_network_is_updated_on_new_rule_creation(self): + sheerka, context = self.init_test().unpack() + evaluate_rule_service = sheerka.services[SheerkaEvaluateRules.NAME] + rule_manager_service = sheerka.services[SheerkaRuleManager.NAME] + + rule = Rule(ACTION_TYPE_EXEC, "test", "ret.status == True", "test()") + rule_manager_service.init_rule(context, rule) + sheerka.create_new_rule(context, rule) + + assert rule in evaluate_rule_service.network.rules + assert rule.rete_net == evaluate_rule_service.network diff --git a/tests/core/test_SheerkaEventManager.py b/tests/core/test_SheerkaEventManager.py index ff4496c..fdf8774 100644 --- a/tests/core/test_SheerkaEventManager.py +++ b/tests/core/test_SheerkaEventManager.py @@ -80,6 +80,6 @@ example_of_class_method. event=xxx, data='42' assert "my first topic" in service.subscribers assert "my second topic" in service.subscribers - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert "my first topic" in service.subscribers assert "my second topic" not in service.subscribers diff --git a/tests/core/test_SheerkaFunctionsParametersHistory.py b/tests/core/test_SheerkaFunctionsParametersHistory.py index ff18697..d67dece 100644 --- a/tests/core/test_SheerkaFunctionsParametersHistory.py +++ b/tests/core/test_SheerkaFunctionsParametersHistory.py @@ -122,7 +122,7 @@ class TestSheerkaFunctionsParametersHistory(TestUsingMemoryBasedSheerka): 3: [("'string value'", 1)] })} - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.om.copy(service.FUNCTIONS_PARAMETERS_ENTRY) == {"function": FunctionParametersObj( context.event.get_digest(), "function", diff --git a/tests/core/test_SheerkaIsAManager.py b/tests/core/test_SheerkaIsAManager.py index baac637..2aa09b2 100644 --- a/tests/core/test_SheerkaIsAManager.py +++ b/tests/core/test_SheerkaIsAManager.py @@ -322,7 +322,7 @@ class TestSheerkaIsAManager(TestUsingMemoryBasedSheerka): assert sheerka.isa(foo, group2) # I can revert back - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.isaset(context, group1) assert sheerka.isinset(foo, group1) assert sheerka.isa(foo, group1) diff --git a/tests/core/test_SheerkaRuleManager.py b/tests/core/test_SheerkaRuleManager.py index 57c9d0c..16d1e18 100644 --- a/tests/core/test_SheerkaRuleManager.py +++ b/tests/core/test_SheerkaRuleManager.py @@ -3,15 +3,21 @@ import ast import pytest from core.builtin_concepts import BuiltinConcepts -from core.concept import Concept, CMV -from core.global_symbols import RULE_COMPARISON_CONTEXT +from core.concept import Concept, CMV, DEFINITION_TYPE_DEF, CC, DoNotResolve +from core.global_symbols import RULE_COMPARISON_CONTEXT, NotFound, EVENT_RULE_DELETED from core.rule import Rule, ACTION_TYPE_PRINT, ACTION_TYPE_EXEC -from core.sheerka.services.SheerkaRuleManager import SheerkaRuleManager, FormatRuleParser, \ +from core.sheerka.Sheerka import RECOGNIZED_BY_ID, RECOGNIZED_BY_NAME +from core.sheerka.services.SheerkaRuleManager import SheerkaRuleManager, FormatRuleActionParser, \ FormatAstRawText, FormatAstVariable, FormatAstSequence, FormatAstFunction, \ - FormatRuleSyntaxError, FormatAstList, UnexpectedEof, FormatAstColor, RulePredicate, FormatAstDict, FormatAstMulti + FormatRuleSyntaxError, FormatAstList, UnexpectedEof, FormatAstColor, RuleCompiledPredicate, FormatAstDict, \ + FormatAstMulti, \ + PythonCodeEmitter, NoConditionFound, FormatAstNode +from core.sheerka.services.sheerka_service import FailedToCompileError from core.tokenizer import Token, TokenKind from parsers.BaseNodeParser import SourceCodeWithConceptNode, SourceCodeNode from parsers.PythonParser import PythonNode +from sheerkarete.common import V +from sheerkarete.conditions import Condition, AndConditions from tests.TestUsingFileBasedSheerka import TestUsingFileBasedSheerka from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka @@ -32,7 +38,7 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): (ACTION_TYPE_EXEC, SheerkaRuleManager.EXEC_RULE_ENTRY), ]) def test_i_can_create_a_new_rule(self, action_type, cache_entry): - sheerka, context = self.init_concepts(cache_only=False) + sheerka, context = self.init_test(cache_only=False).unpack() previous_rules_number = sheerka.om.get_all(sheerka.OBJECTS_IDS_ENTRY)[SheerkaRuleManager.RULE_IDS] rule = Rule(action_type, "name", "True", "Hello world") @@ -96,6 +102,114 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): sheerka.om.current_cache_manager().caches[ SheerkaRuleManager.FORMAT_RULE_ENTRY].cache) == 2 + previous_rules_number + @pytest.mark.parametrize("action_type, cache_entry", [ + (ACTION_TYPE_PRINT, SheerkaRuleManager.FORMAT_RULE_ENTRY), + (ACTION_TYPE_EXEC, SheerkaRuleManager.EXEC_RULE_ENTRY), + ]) + def test_i_can_delete_a_rule(self, action_type, cache_entry): + sheerka, context, rule = self.init_test(cache_only=False).with_rules( + action_type, + Rule(action_type, "rule_name", "id.attr == 'value'", 'True')).unpack() + rule_to_delete = Rule(rule.compiled_action, rule_id=rule.id) + + event_sink = [] + self.sheerka.subscribe(EVENT_RULE_DELETED, lambda c, r: event_sink.append(r)) + + ret = sheerka.remove_rule(context, rule_to_delete) + + assert ret.status + assert sheerka.om.get(cache_entry, rule.id) is NotFound + assert sheerka.om.get(SheerkaRuleManager.RULES_BY_NAME_ENTRY, rule.id) is NotFound + + sheerka.om.commit(context) + assert sheerka.om.current_sdp().get(cache_entry, rule.id) + assert sheerka.om.current_sdp().get(SheerkaRuleManager.RULES_BY_NAME_ENTRY, rule.id) + + assert event_sink == [rule] + + def test_i_can_init_rule_with_a_exec_rule(self): + sheerka, context = self.init_test(cache_only=False).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + rule = Rule(ACTION_TYPE_EXEC, "name", "__input == 'hello world'", "'Hello back at you !'") + rule = service.init_rule(context, rule) + + assert rule.metadata.is_compiled + assert rule.metadata.is_enabled + assert len(rule.compiled_predicates) == 1 + assert len(rule.rete_disjunctions) == 1 + assert sheerka.isinstance(rule.compiled_action, BuiltinConcepts.RETURN_VALUE) + assert rule.compiled_action.status + assert rule.error_sink is None + + def test_i_can_init_rule_with_a_format_rule(self): + sheerka, context = self.init_test(cache_only=False).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + rule = Rule(ACTION_TYPE_PRINT, "name", "__input == 'hello world'", "Hello back at you !") + rule = service.init_rule(context, rule) + + assert rule.metadata.is_compiled + assert rule.metadata.is_enabled + assert len(rule.compiled_predicates) == 1 + assert len(rule.rete_disjunctions) == 1 + assert isinstance(rule.compiled_action, FormatAstNode) + assert rule.error_sink is None + + def test_i_do_not_init_rule_an_already_compiled_rule(self): + sheerka, context = self.init_test(cache_only=False).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + rule = Rule(ACTION_TYPE_EXEC, "name", "cannot build", "cannot compile either") + rule.metadata.is_compiled = True + + rule = service.init_rule(context, rule) + + assert rule.metadata.is_compiled + assert rule.error_sink is None # no error detected + + @pytest.mark.parametrize("action_type, action", [ + (ACTION_TYPE_EXEC, "'Hello back at you !'"), + (ACTION_TYPE_PRINT, "Hello back at you !"), + ]) + def test_init_rule_returns_errors_when_cannot_build_predicate(self, action_type, action): + sheerka, context = self.init_test(cache_only=False).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + rule = Rule(ACTION_TYPE_EXEC, "name", "cannot build", "'Hello back at you !'") + rule.metadata.is_enabled = True # it should be disabled + rule = service.init_rule(context, rule) + + assert len(rule.error_sink["when"]) > 0 + assert sheerka.is_error(rule.error_sink["when"][0]) + assert "print" not in rule.error_sink + assert "then" not in rule.error_sink + assert rule.metadata.is_compiled + assert not rule.metadata.is_enabled + assert rule.compiled_predicates is None + assert rule.rete_disjunctions is None + + @pytest.mark.parametrize("action_type, action", [ + (ACTION_TYPE_EXEC, "cannot build action"), + (ACTION_TYPE_PRINT, "list("), + ]) + def test_init_rule_returns_error_when_cannot_build_action(self, action_type, action): + sheerka, context = self.init_test(cache_only=False).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + rule = Rule(action_type, "name", "__input == 'hello world'", action) + rule.metadata.is_enabled = True # it should be disabled + rule = service.init_rule(context, rule) + + other_action_type = ACTION_TYPE_PRINT if action_type == ACTION_TYPE_EXEC else ACTION_TYPE_EXEC + + assert sheerka.is_error(rule.error_sink[action_type]) + assert other_action_type not in rule.error_sink + assert "when" not in rule.error_sink + assert rule.metadata.is_compiled + assert not rule.metadata.is_enabled + assert rule.compiled_action is None + @pytest.mark.parametrize("text, expected", [ ("", FormatAstRawText("")), (" ", FormatAstRawText(" ")), @@ -134,7 +248,7 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): ("multi(var_name)", FormatAstMulti("var_name")), ]) def test_i_can_parse_format_rule(self, text, expected): - assert FormatRuleParser(text).parse() == expected + assert FormatRuleActionParser(text).parse() == expected @pytest.mark.parametrize("text, expected_error", [ ("{", UnexpectedEof("while parsing variable", Token(TokenKind.LBRACE, "{", 0, 1, 1))), @@ -156,15 +270,52 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): ("dict()", FormatRuleSyntaxError("variable name not found", None)), ]) def test_i_cannot_parse_invalid_format(self, text, expected_error): - parser = FormatRuleParser(text) + parser = FormatRuleActionParser(text) parser.parse() assert parser.error_sink == expected_error + @pytest.mark.parametrize("text", [ + "__action == 'some_action' and True", + "__action == 'some_action' and not a question", + "__action == 'some_action' and is a question", + ]) + def test_i_can_compile_predicate_when_action_is_provided(self, text): + sheerka, context, *concepts = self.init_test().with_concepts( + "a question", + Concept("is a question", pre='is_question()'), + create_new=True).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].action == 'some_action' + + def test_i_can_compile___action_is_not_part_of_the_predicates(self): + sheerka, context, *concepts = self.init_concepts("foo") + service = sheerka.services[SheerkaRuleManager.NAME] + ast_ = ast.parse("a == 5", "", 'eval') + expected_python_node = PythonNode('a == 5', ast_) + + compiled_result = service.compile_when(context, "test", "__action == 'some action' and a == 5") + res = compiled_result.compiled_predicates + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert sheerka.objvalue(res[0].predicate) == expected_python_node + assert res[0].concept is None + @pytest.mark.parametrize("text", [ "a == 5", - "foo == 5", + "foo > 5", "func() == 5", + "not a == 5", + "not foo > 5", + "not func() == 5", ]) def test_i_can_compile_predicate_when_pure_python(self, text): sheerka, context, *concepts = self.init_concepts("foo") @@ -172,10 +323,11 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): ast_ = ast.parse(text, "", 'eval') expected_python_node = PythonNode(text, ast_) - res = service.compile_when(context, "test", text) + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates assert len(res) == 1 - assert isinstance(res[0], RulePredicate) + assert isinstance(res[0], RuleCompiledPredicate) assert res[0].evaluator == PYTHON_EVALUATOR_NAME assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) assert sheerka.objvalue(res[0].predicate) == expected_python_node @@ -184,6 +336,8 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): @pytest.mark.parametrize("text, expected_type", [ ("isinstance(a, int)", SourceCodeWithConceptNode), ("func()", SourceCodeNode), + ("not isinstance(a, int)", PythonNode), + ("not func()", PythonNode), ]) def test_i_can_compile_predicates_that_resolve_to_python(self, text, expected_type): sheerka, context, *concepts = self.init_concepts() @@ -191,30 +345,15 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): ast_ = ast.parse(text, "", 'eval') expected_python_node = PythonNode(text, ast_) - res = service.compile_when(context, "test", text) + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates assert len(res) == 1 - assert isinstance(res[0], RulePredicate) + assert isinstance(res[0], RuleCompiledPredicate) assert res[0].evaluator == PYTHON_EVALUATOR_NAME assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) assert isinstance(sheerka.objvalue(res[0].predicate), expected_type) - assert sheerka.objvalue(res[0].predicate).python_node == expected_python_node - assert res[0].concept is None - - def test_i_can_compile_predicate_when_python_and_concept(self): - sheerka, context, *concepts = self.init_test().with_concepts(Concept("foo bar"), create_new=True).unpack() - service = sheerka.services[SheerkaRuleManager.NAME] - text = "foo bar == 5" - ast_ = ast.parse("__C__foo0bar__1001__C__ == 5", "", 'eval') - resolved_expected = PythonNode(text, ast_) - - res = service.compile_when(context, "test", text) - - assert len(res) == 1 - assert isinstance(res[0], RulePredicate) - assert res[0].evaluator == PYTHON_EVALUATOR_NAME - assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) - assert sheerka.objvalue(res[0].predicate) == resolved_expected + assert sheerka.objvalue(res[0].predicate).get_python_node() == expected_python_node assert res[0].concept is None @pytest.mark.parametrize("text, expected_variables", [ @@ -234,15 +373,106 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): expected = concepts[0] expected.get_metadata().variables = [('x', expected_variables[0]), ('y', expected_variables[1])] - res = service.compile_when(context, "test", text) + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates assert len(res) == 1 - assert isinstance(res[0], RulePredicate) + assert isinstance(res[0], RuleCompiledPredicate) assert res[0].evaluator == CONCEPT_EVALUATOR_NAME assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) assert sheerka.objvalue(res[0].predicate) == expected assert res[0].concept == expected + @pytest.mark.parametrize("text, text_to_compile, expected_variables", [ + ("not cat is an animal", "not __C__00var0000is0an000var001__1001__C__", ["cat", "animal"]), + ("not a is an animal", "not __C__00var0000is0an000var001__1001__C__", ["a", "animal"]), + ("not cat is an b", "not __C__00var0000is0an000var001__1001__C__", ["cat", "b"]), + ]) + def test_i_can_compile_negative_predicate_when_exact_concept(self, text, text_to_compile, expected_variables): + sheerka, context, *concepts = self.init_test().with_concepts( + Concept("x is an y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("cat"), + Concept("animal"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + expected_concept = concepts[0] + expected_concept.get_metadata().variables = [('x', expected_variables[0]), ('y', expected_variables[1])] + ast_ = ast.parse(text_to_compile, "", 'eval') + expected_python_node = PythonNode(text_to_compile, ast_) + expected_python_node.original_source = text + expected_python_node.objects = {"__C__00var0000is0an000var001__1001__C__": expected_concept} + res = service.compile_when(context, "test", text) + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert isinstance(sheerka.objvalue(res[0].predicate), PythonNode) + + python_node = sheerka.objvalue(res[0].predicate).get_python_node() + assert python_node == expected_python_node + assert len(python_node.objects) == 1 + assert python_node.objects["__C__00var0000is0an000var001__1001__C__"] == expected_concept + assert res[0].concept is None + + @pytest.mark.skip("Not managed yet") + @pytest.mark.parametrize("text, text_to_compile, expected_variables", [ + ("not cat is an animal", "__C__00var0000is0an000var001__1001__C__", ["not cat", "animal"]) + ]) + def test_i_can_compile_negative_predicate_when_exact_concept(self, text, text_to_compile, expected_variables): + sheerka, context, *concepts = self.init_test().with_concepts( + Concept("x is an y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("cat"), + Concept("animal"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + expected_concept = concepts[0] + expected_concept.get_metadata().variables = [('x', expected_variables[0]), ('y', expected_variables[1])] + ast_ = ast.parse(text_to_compile, "", 'eval') + expected_python_node = PythonNode(text_to_compile, ast_) + expected_python_node.original_source = text + expected_python_node.objects = {"__C__00var0000is0an000var001__1001__C__": expected_concept} + res = service.compile_when(context, "test", text) + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert isinstance(sheerka.objvalue(res[0].predicate), PythonNode) + + python_node = sheerka.objvalue(res[0].predicate).get_python_node() + assert python_node == expected_python_node + assert len(python_node.objects) == 1 + assert python_node.objects["__C__00var0000is0an000var001__1001__C__"] == expected_concept + assert res[0].concept is None + + @pytest.mark.parametrize("text, text_to_compile", [ + ("foo bar == 5", "__C__foo0bar__1001__C__ == 5"), + ("not foo bar == 5", "not __C__foo0bar__1001__C__ == 5"), + ]) + def test_i_can_compile_predicate_when_python_and_concept(self, text, text_to_compile): + sheerka, context, *concepts = self.init_test().with_concepts(Concept("foo bar"), create_new=True).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + ast_ = ast.parse(text_to_compile, "", 'eval') + resolved_expected = PythonNode(text_to_compile, ast_, text) + + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + + python_node = sheerka.objvalue(res[0].predicate).get_python_node() + assert python_node == resolved_expected + assert python_node.objects == {'__C__foo0bar__1001__C__': concepts[0]} + assert res[0].concept is None + @pytest.mark.parametrize("text, expected_variables", [ ("a cat is an animal", ["a cat", "animal"]), ("a cat is an b", ["a cat", "b"]), @@ -257,10 +487,11 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): service = sheerka.services[SheerkaRuleManager.NAME] expected = CMV(concepts[0], x=expected_variables[0], y=expected_variables[1]) - res = service.compile_when(context, "test", text) + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates assert len(res) == 1 - assert isinstance(res[0], RulePredicate) + assert isinstance(res[0], RuleCompiledPredicate) assert res[0].evaluator == CONCEPT_EVALUATOR_NAME assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) assert sheerka.objvalue(res[0].predicate)[0].concept == expected @@ -275,15 +506,101 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): service = sheerka.services[SheerkaRuleManager.NAME] expected = concepts[1] - res = service.compile_when(context, "test", "cat is an animal") + compiled_result = service.compile_when(context, "test", "cat is an animal") + res = compiled_result.compiled_predicates assert len(res) == 1 - assert isinstance(res[0], RulePredicate) + assert isinstance(res[0], RuleCompiledPredicate) assert res[0].evaluator == CONCEPT_EVALUATOR_NAME assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) assert sheerka.objvalue(res[0].predicate)[0].concept == expected assert res[0].concept == expected + def test_i_can_compile_predicate_when_mix_of_concepts_and_python(self): + sheerka, context, animal, cat, dog, pet, is_a, is_an = self.init_test().with_concepts( + Concept("animal"), + Concept("a cat"), + Concept("dog"), + Concept("pet"), + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is an y", pre="is_question()", definition="('cat'|'bird')=x 'is an' animal").def_var("x"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + text = "a cat is a pet and bird is an animal and x > 5 and dog is a pet" + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + to_compile = '__C__00var0000is0a000var001__1005__C__ and __C__00var0000is0an0y__1006__C__ and x > 5 and __C__00var0000is0a000var001__1005_1__C__' + ast_ = ast.parse(to_compile, "", 'eval') + expected_python_node = PythonNode(to_compile, ast_, text) + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + python_node = res[0].predicate.body.body + assert python_node == expected_python_node + assert python_node.objects == { + "__C__00var0000is0a000var001__1005__C__": CC(is_a, x=cat, y=pet), + "__C__00var0000is0an0y__1006__C__": CC(is_an, exclude_body=True, x=DoNotResolve("bird"), animal=animal), + "__C__00var0000is0a000var001__1005_1__C__": CMV(is_a, x="dog", y="pet"), + } + + @pytest.mark.parametrize("text", [ + "a and not b", + "not b and a", + "__ret and not __ret.status", + ]) + def test_i_can_compile_negative_conjunctions_when_pure_python(self, text): + sheerka, context, *concepts = self.init_concepts("foo") + service = sheerka.services[SheerkaRuleManager.NAME] + + ast_ = ast.parse(text, "", 'eval') + expected_python_node = PythonNode(text, ast_) + + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert sheerka.objvalue(res[0].predicate) == expected_python_node + assert res[0].concept is None + + def test_i_can_compile_negative_conjunction_of_mix_of_concepts_and_python(self): + sheerka, context, animal, cat, dog, pet, is_a, is_an = self.init_test().with_concepts( + Concept("animal"), + Concept("a cat"), + Concept("dog"), + Concept("pet"), + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is an y", pre="is_question()", definition="('cat'|'bird')=x 'is an' animal").def_var("x"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + text = "not a cat is a pet and not bird is an animal and not x > 5 and not dog is a pet" + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + to_compile = 'not __C__00var0000is0a000var001__1005__C__' + to_compile += ' and not __C__00var0000is0an0y__1006__C__' + to_compile += ' and not x > 5' + to_compile += ' and not __C__00var0000is0a000var001__1005_1__C__' + ast_ = ast.parse(to_compile, "", 'eval') + expected_python_node = PythonNode(to_compile, ast_, text) + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + python_node = res[0].predicate.body.body + assert python_node == expected_python_node + assert python_node.objects == { + "__C__00var0000is0a000var001__1005__C__": CC(is_a, x=cat, y=pet), + "__C__00var0000is0an0y__1006__C__": CC(is_an, exclude_body=True, x=DoNotResolve("bird"), animal=animal), + "__C__00var0000is0a000var001__1005_1__C__": CMV(is_a, x="dog", y="pet"), + } + def test_i_can_compile_predicate_when_multiple_choices(self): sheerka, context, *concepts = self.init_test().with_concepts( Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), @@ -292,24 +609,185 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): ).unpack() service = sheerka.services[SheerkaRuleManager.NAME] - res = service.compile_when(context, "test", "a is a b") + compiled_result = service.compile_when(context, "test", "a is a b") + res = compiled_result.compiled_predicates assert len(res) == 2 - assert isinstance(res[0], RulePredicate) + assert isinstance(res[0], RuleCompiledPredicate) assert res[0].evaluator == CONCEPT_EVALUATOR_NAME assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) assert sheerka.objvalue(res[0].predicate)[0].concept == CMV(concepts[0], x="a", y="b") assert res[0].concept == CMV(concepts[0], x="a", y="b") - assert isinstance(res[1], RulePredicate) + assert isinstance(res[1], RuleCompiledPredicate) assert res[1].evaluator == CONCEPT_EVALUATOR_NAME assert sheerka.isinstance(res[1].predicate, BuiltinConcepts.RETURN_VALUE) assert sheerka.objvalue(res[1].predicate)[0].concept == CMV(concepts[1], x="a", y="b") assert res[1].concept == CMV(concepts[1], x="a", y="b") + def test_i_can_compile_predicate_when_mix_and_multiple_choices(self): + sheerka, context, *concepts = self.init_test().with_concepts( + Concept("animal"), + Concept("a cat"), + Concept("dog"), + Concept("pet"), + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is a y", pre="is_question()", body="isa(x, y)").def_var("x").def_var("y"), + Concept("x is an y", pre="is_question()", definition="('cat'|'bird')=x 'is an' animal").def_var("x"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + text = "__action == 'value' and a cat is a pet and bird is an animal and x > 5 and dog is a pet" + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + assert len(res) == 4 + trimmed_source = "a cat is a pet and bird is an animal and x > 5 and dog is a pet" + + current_res = res[0] + assert isinstance(current_res, RuleCompiledPredicate) + assert current_res.evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(current_res.predicate, BuiltinConcepts.RETURN_VALUE) + python_source = "__C__00var0000is0a000var001__1005__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1005_1__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_res.predicate) == resolved_expected + assert current_res.concept is None + + current_res = res[1] + assert isinstance(current_res, RuleCompiledPredicate) + assert current_res.evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(current_res.predicate, BuiltinConcepts.RETURN_VALUE) + python_source = "__C__00var0000is0a000var001__1005__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1006__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_res.predicate) == resolved_expected + assert current_res.concept is None + + current_res = res[2] + assert isinstance(current_res, RuleCompiledPredicate) + assert current_res.evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(current_res.predicate, BuiltinConcepts.RETURN_VALUE) + python_source = "__C__00var0000is0a000var001__1006__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1005__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_res.predicate) == resolved_expected + assert current_res.concept is None + + current_res = res[3] + assert isinstance(current_res, RuleCompiledPredicate) + assert current_res.evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(current_res.predicate, BuiltinConcepts.RETURN_VALUE) + python_source = "__C__00var0000is0a000var001__1006__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1006_1__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_res.predicate) == resolved_expected + assert current_res.concept is None + + @pytest.mark.parametrize("text, mode, compiled_text", [ + ("greetings", "eval", f"__ret.status and isinstance(__ret.body, Concept) and __ret.body.name == 'greetings'"), + ("c:|1001:", "eval", f"__ret.status and isinstance(__ret.body, Concept) and __ret.body.id == '1001'"), + ("hello 'there'", "eval", + f"__ret.status and isinstance(__ret.body, Concept) and __ret.body.key == 'hello __var__0' and __ret.body.get_value('a') == 'there'"), + ("hello there", "exec", + f"__x_00__ = __ret.body.get_value('a')\n__ret.status and isinstance(__ret.body, Concept) and __ret.body.key == 'hello __var__0' and isinstance(__x_00__, Concept) and __x_00__.key == 'there'"), + ("hello my friend", "exec", + f"__x_00__ = __ret.body.get_value('a')\n__ret.status and isinstance(__ret.body, Concept) and __ret.body.key == 'hello __var__0' and isinstance(__x_00__, Concept) and __x_00__.key == 'my friend'"), + ]) + def test_i_can_compile_predicate_when_concept_is_not_a_question(self, text, mode, compiled_text): + sheerka, context, greetings, there, my_friend = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + Concept("there"), + Concept("my friend"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + ast_ = ast.parse(compiled_text, "", mode) + expected_python_node = PythonNode(compiled_text, ast_) + + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert sheerka.objvalue(res[0].predicate) == expected_python_node + assert res[0].concept is None + + def test_i_can_compile_predicate_when_concept_is_not_a_question_and_involves_sheerka(self): + sheerka, context, greetings = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + compiled_text = "__x_00__ = __ret.body.get_value('a')\n" + compiled_text += "__ret.status" + compiled_text += " and isinstance(__ret.body, Concept) and __ret.body.key == 'hello __var__0'" + compiled_text += " and isinstance(__x_00__, Expando) and __x_00__.get_name() == 'sheerka'" + ast_ = ast.parse(compiled_text, "", "exec") + expected_python_node = PythonNode(compiled_text, ast_) + + compiled_result = service.compile_when(context, "test", "hello sheerka") + res = compiled_result.compiled_predicates + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == PYTHON_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert sheerka.objvalue(res[0].predicate) == expected_python_node + assert res[0].concept is None + + @pytest.mark.skip("Not managed yet") + def test_i_can_compile_when_concept_starts_with_not(self): + sheerka, context, *concepts = self.init_test().with_concepts( + Concept("not a cheesecake", pre="is_question()"), + create_new=True).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + text = "not a cheesecake" + expected = concepts[0] + + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.compiled_predicates + + assert len(res) == 1 + assert isinstance(res[0], RuleCompiledPredicate) + assert res[0].evaluator == CONCEPT_EVALUATOR_NAME + assert sheerka.isinstance(res[0].predicate, BuiltinConcepts.RETURN_VALUE) + assert sheerka.objvalue(res[0].predicate) == expected + assert res[0].concept == expected + + def test_i_cannot_compile_when_concept_is_not_a_question_and_has_unknown_variable(self): + sheerka, context, greetings = self.init_test().with_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + create_new=True + ).unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + with pytest.raises(FailedToCompileError) as ex: + service.compile_when(context, "test", "hello there") + + assert sheerka.isinstance(ex.value.cause[0], BuiltinConcepts.CONCEPT_EVAL_ERROR) + + @pytest.mark.parametrize("text, expected_error", [ + ("__action == 'some_action'", NoConditionFound()) + ]) + def test_i_cannot_compile_when_error(self, text, expected_error): + sheerka, context = self.init_test().unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + with pytest.raises(FailedToCompileError) as ex: + service.compile_when(context, "test", text) + + assert ex.value.cause == [expected_error] + def test_i_can_get_rule_priorities(self): - sheerka, context, rule_true, rule_false = self.init_test().with_rules(("True", "True"), - ("False", "False")).unpack() + sheerka, context, rule_true, rule_false = self.init_test().with_format_rules(("True", "True"), + ("False", "False")).unpack() sheerka.set_is_greater_than(context, BuiltinConcepts.PRECEDENCE, rule_true, @@ -322,7 +800,7 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): assert rules_from_cache[rule_false.id].priority == 1 def test_i_can_get_and_retrieve_rules_when_multiple_ontology_layers(self): - sheerka, context, rule_true = self.init_test().with_rules(("true", "True", "True")).unpack() + sheerka, context, rule_true = self.init_test().with_format_rules(("true", "True", "True")).unpack() sheerka.push_ontology(context, "new ontology") rule_false = sheerka.create_new_rule(context, Rule(ACTION_TYPE_EXEC, "false", "False", "False")).body.body @@ -331,12 +809,12 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): assert sheerka.get_rule_by_id(rule_true.id) == rule_true assert sheerka.get_rule_by_id(rule_false.id) == rule_false - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.get_rule_by_id(rule_true.id) == rule_true assert not sheerka.is_known(sheerka.get_rule_by_id(rule_false.id)) def test_i_can_resolve_rule(self): - sheerka, context, rule = self.init_test().with_rules(("my rule", "True", "True")).unpack() + sheerka, context, rule = self.init_test().with_format_rules(("my rule", "True", "True")).unpack() context.add_to_short_term_memory("x", rule.id) # direct access by id @@ -370,57 +848,175 @@ class TestSheerkaRuleManager(TestUsingMemoryBasedSheerka): unresolved.metadata.id_is_unresolved = True assert sheerka.resolve_rule(context, unresolved) == rule - # @pytest.mark.skip - # @pytest.mark.parametrize("text, expected", [ - # ("cat is an animal", set()), - # ("a is an animal", {"a"}), - # ("a is an b", {"a", "b"}), - # ("cat is an b", {"b"}), - # ("a cat is an b", {"b"}), - # - # ("cat is a animal", set()), - # ("a is a animal", {"a"}), - # ("a is a b", {"a", "b"}), - # ("cat is a b", {"b"}), - # ("a cat is an b", {"b"}), - # - # ("a == 5", {"a"}), - # ("isinstance(a, int)", {"a"}), - # ("a cat == b", {"b"}) - # ]) - # def test_i_can_get_rules_variables(self, text, expected): - # sheerka, context, *concepts = self.init_concepts( - # Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), - # Concept("x is a y", pre="is_question()", body="isa(x, y)").def_var("x").def_var("y"), - # Concept("x is an y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), - # Concept("cat"), - # Concept("animal"), - # Concept("a cat"), - # create_new=True - # ) - # service = sheerka.services[SheerkaRuleManager.NAME] - # - # compiled = service.compile_when(context, "test", "a is a b") - # - # assert service.get_unknown_variables(compiled) == expected + @pytest.mark.parametrize("obj, expected", [ + ("text value", "var == 'text value'"), + ("text 'value'", '''var == "text 'value'"'''), + ('text "value"', """var == 'text "value"'"""), + (10, "var == 10"), + (10.01, "var == 10.01"), + ]) + def test_i_can_test_python_code_emitter_for_basic_types(self, obj, expected): + sheerka, context = self.init_test().unpack() + + assert PythonCodeEmitter(context).recognize(obj, "var").get_text() == expected + assert PythonCodeEmitter(context, "status").recognize(obj, "var").get_text() == "status and " + expected + + @pytest.mark.parametrize("recognized_by, expected", [ + (RECOGNIZED_BY_ID, "isinstance(var, Concept) and var.id == '1001'"), + (RECOGNIZED_BY_NAME, "isinstance(var, Concept) and var.name == 'greetings'"), + (None, "isinstance(var, Concept) and var.key == 'hello'"), + ]) + def test_i_can_test_python_code_emitter_for_concepts(self, recognized_by, expected): + sheerka, context, foo = self.init_concepts( + Concept("greetings", definition="hello", definition_type=DEFINITION_TYPE_DEF)) + + instance = sheerka.new_from_template(foo, foo.key) + if recognized_by: + instance.set_hint(BuiltinConcepts.RECOGNIZED_BY, recognized_by) + + assert PythonCodeEmitter(context).recognize(instance, "var").get_text() == expected + assert PythonCodeEmitter(context, "status").recognize(instance, "var").get_text() == "status and " + expected + + def test_i_can_test_python_code_emitter_for_concepts_with_variable(self): + sheerka, context, greetings, little, foo, bar, and_concept = self.init_concepts( + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a"), + Concept("little x").def_var("x"), + "foo", + "bar", + Concept("a and b").def_var("a").def_var("b") + ) + + # variable is a string + greetings_instance = sheerka.new_from_template(greetings, greetings.key, a='sheerka') + expected = "isinstance(var, Concept) and var.key == 'hello __var__0' and var.get_value('a') == 'sheerka'" + text = PythonCodeEmitter(context).recognize(greetings_instance, "var").get_text() + assert text == expected + + # variable is a concept recognized by id + foo_instance = sheerka.new_from_template(foo, foo.key) + foo_instance.set_hint(BuiltinConcepts.RECOGNIZED_BY, RECOGNIZED_BY_ID) + greetings_instance = sheerka.new_from_template(greetings, greetings.key, a=foo_instance) + expected = """__x_00__ = var.get_value('a') +isinstance(var, Concept) and var.key == 'hello __var__0' and isinstance(__x_00__, Concept) and __x_00__.id == '1003'""" + text = PythonCodeEmitter(context).recognize(greetings_instance, "var").get_text() + assert text == expected + + # variable is a concept recognized by name + foo_instance = sheerka.new_from_template(foo, foo.key) + foo_instance.set_hint(BuiltinConcepts.RECOGNIZED_BY, RECOGNIZED_BY_NAME) + greetings_instance = sheerka.new_from_template(greetings, greetings.key, a=foo_instance) + expected = """__x_00__ = var.get_value('a') +isinstance(var, Concept) and var.key == 'hello __var__0' and isinstance(__x_00__, Concept) and __x_00__.name == 'foo'""" + text = PythonCodeEmitter(context).recognize(greetings_instance, "var").get_text() + assert text == expected + + # variable is a concept recognized by value + foo_instance = sheerka.new_from_template(foo, foo.key) + greetings_instance = sheerka.new_from_template(greetings, greetings.key, a=foo_instance) + expected = """__x_00__ = var.get_value('a') +isinstance(var, Concept) and var.key == 'hello __var__0' and isinstance(__x_00__, Concept) and __x_00__.key == 'foo'""" + text = PythonCodeEmitter(context).recognize(greetings_instance, "var").get_text() + assert text == expected + + # variable is a concept witch has itself some variable + foo_instance = sheerka.new_from_template(foo, foo.key) + little_instance = sheerka.new_from_template(little, little.key, x=foo_instance) + greetings_instance = sheerka.new_from_template(greetings, greetings.key, a=little_instance) + expected = """__x_00__ = var.get_value('a') +__x_01__ = __x_00__.get_value('x') +isinstance(var, Concept) and var.key == 'hello __var__0'""" + \ + " and isinstance(__x_00__, Concept) and __x_00__.key == 'little __var__0'" + \ + " and isinstance(__x_01__, Concept) and __x_01__.key == 'foo'""" + text = PythonCodeEmitter(context).recognize(greetings_instance, "var").get_text() + assert text == expected + + # concept with multiple variables (which are themselves concepts) + foo_instance = sheerka.new_from_template(foo, foo.key) + bar_instance = sheerka.new_from_template(bar, bar.key) + little_instance = sheerka.new_from_template(little, little.key, x=foo_instance) + and_instance = sheerka.new_from_template(and_concept, and_concept.key, a=bar_instance, b=little_instance) + greetings_instance = sheerka.new_from_template(greetings, greetings.key, a=and_instance) + expected = """__x_00__ = var.get_value('a') +__x_01__ = __x_00__.get_value('a') +__x_02__ = __x_00__.get_value('b') +__x_03__ = __x_02__.get_value('x') +isinstance(var, Concept) and var.key == 'hello __var__0'""" + \ + " and isinstance(__x_00__, Concept) and __x_00__.key == '__var__0 and __var__1'" + \ + " and isinstance(__x_01__, Concept) and __x_01__.key == 'bar'" + \ + " and isinstance(__x_02__, Concept) and __x_02__.key == 'little __var__0'" + \ + " and isinstance(__x_03__, Concept) and __x_03__.key == 'foo'" + text = PythonCodeEmitter(context).recognize(greetings_instance, "var").get_text() + assert text == expected + + def test_i_can_get_format_rules(self): + sheerka, context = self.init_test().unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + sheerka.om.current_cache_manager().clear(service.FORMAT_RULE_ENTRY) + r1 = Rule(ACTION_TYPE_PRINT, "name 1", "True", "Hello world 1", priority=1) + r2 = Rule(ACTION_TYPE_PRINT, "name 2", "False", "Hello world 2", priority=3) + r3 = Rule(ACTION_TYPE_PRINT, "name 2", "None", "Hello world 3", priority=2) + sheerka.create_new_rule(context, r1) + sheerka.create_new_rule(context, r2) + sheerka.create_new_rule(context, r3) + + res = sheerka.get_format_rules() + assert res == [r2, r3, r1] + + def test_i_can_get_exec_rules(self): + sheerka, context = self.init_test().unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + sheerka.om.current_cache_manager().clear(service.EXEC_RULE_ENTRY) + r1 = Rule(ACTION_TYPE_EXEC, "name 1", "True", "'Hello world 1'", priority=1) + r2 = Rule(ACTION_TYPE_EXEC, "name 2", "False", "'Hello world 2'", priority=3) + r3 = Rule(ACTION_TYPE_EXEC, "name 2", "None", "'Hello world 3'", priority=2) + sheerka.create_new_rule(context, r1) + sheerka.create_new_rule(context, r2) + sheerka.create_new_rule(context, r3) + + res = sheerka.get_exec_rules() + assert res == [r2, r3, r1] + + def test_i_can_compile_rete_using_name(self): + sheerka, context, *concepts = self.init_test().unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + + text = "__ret" + + compiled_result = service.compile_when(context, "test", text) + res = compiled_result.rete_disjunctions + + assert len(res) == 1 + assert isinstance(res[0], AndConditions) + assert res[0].conditions == [Condition(V("__x_00__"), "__name__", "__ret")] class TestSheerkaRuleManagerUsingFileBasedSheerka(TestUsingFileBasedSheerka): def test_rules_are_initialized_at_startup(self): - sheerka, context, *rules = self.init_format_rules( + sheerka, context, *rules = self.init_test().with_rules( + None, Rule("print", "name1", "True", "Hello world"), - Rule("print", "name2", "value() is __EXPLANATION", "list(value())") - ) + Rule("print", "name2", "value() is __EXPLANATION", "list(value())"), + Rule("exec", "name3", "True", "'Hello world'"), + Rule("exec", "name4", "value() is __EXPLANATION", "list(value())"), + ).unpack() sheerka.set_is_greater_than(context, BuiltinConcepts.PRECEDENCE, rules[0], rules[1], RULE_COMPARISON_CONTEXT) + sheerka.set_is_less_than(context, BuiltinConcepts.PRECEDENCE, + rules[2], + rules[3], + RULE_COMPARISON_CONTEXT) sheerka.om.commit(context) expected_rules_by_id = sheerka.om.get_all(SheerkaRuleManager.FORMAT_RULE_ENTRY) + expected_rules_by_id.update(sheerka.om.get_all(SheerkaRuleManager.EXEC_RULE_ENTRY)) sheerka = self.new_sheerka_instance(False) # new instance rules_by_id = sheerka.om.get_all(SheerkaRuleManager.FORMAT_RULE_ENTRY) + rules_by_id.update(sheerka.om.get_all(SheerkaRuleManager.EXEC_RULE_ENTRY)) assert len(rules_by_id) == len(expected_rules_by_id) @@ -432,10 +1028,10 @@ class TestSheerkaRuleManagerUsingFileBasedSheerka(TestUsingFileBasedSheerka): assert rule.metadata.predicate == expected.metadata.predicate assert rule.metadata.action == expected.metadata.action assert rule.metadata.id == expected.metadata.id - assert rule.metadata.is_compiled == expected.metadata.is_compiled - assert rule.metadata.is_enabled == expected.metadata.is_enabled + assert rule.metadata.is_compiled + assert rule.metadata.is_enabled assert rule.compiled_action == expected.compiled_action - assert rule.compiled_predicate == expected.compiled_predicate + assert rule.compiled_predicates == expected.compiled_predicates assert rule.priority is not None assert rule.priority == expected.priority @@ -468,6 +1064,6 @@ class TestSheerkaRuleManagerUsingFileBasedSheerka(TestUsingFileBasedSheerka): assert rule.metadata.is_compiled == expected.metadata.is_compiled assert rule.metadata.is_enabled == expected.metadata.is_enabled assert rule.compiled_action == expected.compiled_action - assert rule.compiled_predicate == expected.compiled_predicate + assert rule.compiled_predicates == expected.compiled_predicates assert rule.priority is not None assert rule.priority == expected.priority diff --git a/tests/core/test_SheerkaVariableManager.py b/tests/core/test_SheerkaVariableManager.py index 576013e..2e2bf38 100644 --- a/tests/core/test_SheerkaVariableManager.py +++ b/tests/core/test_SheerkaVariableManager.py @@ -110,11 +110,11 @@ class TestSheerkaVariable(TestUsingMemoryBasedSheerka): assert sheerka.load_internal_var("TestSheerkaVariable", "lambda")(10) == 12 # I can revert back - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.load_var("TestSheerkaVariable", "my_variable") == 1 assert sheerka.load_internal_var("TestSheerkaVariable", "lambda")(10) == 11 - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.load_var("TestSheerkaVariable", "my_variable") == 1 assert sheerka.load_internal_var("TestSheerkaVariable", "lambda")(10) == 11 diff --git a/tests/core/test_builtin_helpers.py b/tests/core/test_builtin_helpers.py index 39aa035..883b841 100644 --- a/tests/core/test_builtin_helpers.py +++ b/tests/core/test_builtin_helpers.py @@ -1,10 +1,9 @@ -import ast +import pytest import core.builtin_helpers -import pytest from core.builtin_concepts import ReturnValueConcept, BuiltinConcepts from core.concept import Concept - +from core.global_symbols import NotInit from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka @@ -157,6 +156,32 @@ class TestBuiltinHelpers(TestUsingMemoryBasedSheerka): res = core.builtin_helpers.resolve_ambiguity(context, concepts) assert [c.name for c in res] == expected + @pytest.mark.parametrize("pre, expected", [ + ("x and y", False), + ("is_question()", True), + (" is_question ( ) ", True), + ("context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)", True), + (" context . in_context ( BuiltinConcepts . EVAL_QUESTION_REQUESTED ) ", True), + (None, False), + ("", False), + (NotInit, False), + ("is _ question()", False), + ("is_ question()", False), + ("is _question()", False), + ("context.in _context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)", False), + ("not is_question()", False), + ("not context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)", False), + ("is_question() and True", True), + ("is_question() and False", True), # don't care about the second argument if it is not related to question + ("is_question() and xxx", True), # don't care about the second argument if it is not related to question + ("is_question() and not is_question()", False), # error ? + ("is_question() and not context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)", False), # error ? + ]) + def test_is_a_question(self, pre, expected): + sheerka, context = self.init_test().unpack() + concept = Concept("foo", pre=pre) + assert core.builtin_helpers.is_a_question(context, concept) == expected + # @pytest.mark.parametrize("return_values", [ # None, # [] diff --git a/tests/core/test_sheerka.py b/tests/core/test_sheerka.py index ccf9ea5..07352dc 100644 --- a/tests/core/test_sheerka.py +++ b/tests/core/test_sheerka.py @@ -340,7 +340,7 @@ class TestSheerkaUsingMemoryBasedSheerka(TestUsingMemoryBasedSheerka): sheerka.modify_concept(context, foo, to_add={"variables": {"c": None}}, to_remove={"variables": ["b"]}) assert get_concept_attrs(foo) == ["a", "c"] - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert get_concept_attrs(foo) == ["a", "b"] def test_i_can_manage_concepts_ids_on_multiple_ontology_layers(self): @@ -356,7 +356,7 @@ class TestSheerkaUsingMemoryBasedSheerka(TestUsingMemoryBasedSheerka): res = sheerka.create_new_concept(context, Concept("baz")) assert res.body.body.id == "1003" - sheerka.pop_ontology() + sheerka.pop_ontology(context) res = sheerka.create_new_concept(context, Concept("baz")) assert res.body.body.id == "1002" @@ -380,7 +380,7 @@ class TestSheerkaUsingMemoryBasedSheerka(TestUsingMemoryBasedSheerka): # record the ontology ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() + sheerka.pop_ontology(context) # Create another ontology with some other values sheerka.push_ontology(context, "another ontology") @@ -403,7 +403,7 @@ class TestSheerkaUsingMemoryBasedSheerka(TestUsingMemoryBasedSheerka): assert get_concept_attrs(foo) == ["a", "b"] # sanity check - sheerka.pop_ontology() + sheerka.pop_ontology(context) assert sheerka.get_by_name("foo") == foo2 assert sheerka.get_by_name("bar") == bar assert sheerka.locals == {"key2": "value2"} @@ -562,7 +562,7 @@ class TestSheerkaUsingFileBasedSheerka(TestUsingFileBasedSheerka): sheerka, context = self.init_test().unpack() sheerka.push_ontology(context, "to remove") - sheerka.pop_ontology() + sheerka.pop_ontology(context) sheerka.push_ontology(context, "new ontology") sheerka.push_ontology(context, "another ontology") diff --git a/tests/core/test_sheerkaResultManager.py b/tests/core/test_sheerkaResultManager.py index da26e84..464f311 100644 --- a/tests/core/test_sheerkaResultManager.py +++ b/tests/core/test_sheerkaResultManager.py @@ -21,7 +21,7 @@ class TestSheerkaResultManager(TestUsingMemoryBasedSheerka): @classmethod def teardown_class(cls): - cls.sheerka.pop_ontology() + cls.sheerka.pop_ontology(TestSheerkaResultManager.context) cls.root_ontology_name = SheerkaOntologyManager.ROOT_ONTOLOGY_NAME def init_service(self): @@ -410,7 +410,7 @@ class TestSheerkaResultManagerFileBased(TestUsingFileBasedSheerka): @classmethod def teardown_class(cls): - cls.sheerka.pop_ontology() + cls.sheerka.pop_ontology(TestSheerkaResultManagerFileBased.context) cls.root_ontology_name = SheerkaOntologyManager.ROOT_ONTOLOGY_NAME def test_i_can_retrieve_the_last_error_after_startup(self): diff --git a/tests/core/test_sheerka_ontology.py b/tests/core/test_sheerka_ontology.py index 43fd097..6c8f178 100644 --- a/tests/core/test_sheerka_ontology.py +++ b/tests/core/test_sheerka_ontology.py @@ -8,7 +8,9 @@ from cache.IncCache import IncCache from cache.ListCache import ListCache from cache.ListIfNeededCache import ListIfNeededCache from core.concept import Concept -from core.global_symbols import NotFound, Removed +from core.global_symbols import NotFound, Removed, EVENT_CONCEPT_ID_DELETED, \ + EVENT_RULE_ID_DELETED +from core.rule import Rule, ACTION_TYPE_EXEC from core.sheerka.SheerkaOntologyManager import SheerkaOntologyManager, OntologyManagerFrozen, OntologyManagerNotFrozen, \ OntologyManagerCannotPopLatest, OntologyAlreadyExists from tests.TestUsingFileBasedSheerka import TestUsingFileBasedSheerka @@ -168,22 +170,22 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.current_cache_manager().has("cache_name", "key") def test_i_cannot_pop_ontology_when_not_frozen(self): - sheerka = self.get_sheerka() + sheerka, context = self.init_test().unpack() manager = SheerkaOntologyManager(sheerka, sheerka.root_folder, sheerka.cache_only) with pytest.raises(OntologyManagerNotFrozen): - manager.pop_ontology() + manager.pop_ontology(context) def test_i_cannot_pop_the_latest_cache_manager(self): - sheerka = self.get_sheerka() + sheerka, context = self.init_test().unpack() manager = SheerkaOntologyManager(sheerka, sheerka.root_folder, sheerka.cache_only) manager.freeze() with pytest.raises(OntologyManagerCannotPopLatest): - manager.pop_ontology() + manager.pop_ontology(context) def test_i_can_pop_ontology(self): - sheerka = self.get_sheerka() + sheerka, context = self.init_test().unpack() manager = SheerkaOntologyManager(sheerka, sheerka.root_folder, sheerka.cache_only) manager.freeze() @@ -191,13 +193,13 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): manager.push_ontology("ontology2") manager.push_ontology("ontology3") - manager.pop_ontology() + manager.pop_ontology(context) assert len(manager.ontologies) == 3 - manager.pop_ontology() - manager.pop_ontology() + manager.pop_ontology(context) + manager.pop_ontology(context) with pytest.raises(OntologyManagerCannotPopLatest): - manager.pop_ontology() + manager.pop_ontology(context) def test_i_can_add_ontology(self): sheerka, context = self.init_test().unpack() @@ -217,7 +219,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.get("cache_name", "key3") is NotFound new_ontology = manager.get_ontology() - manager.pop_ontology() + manager.pop_ontology(context) # add another ontology, with its own values manager.push_ontology("another ontology") @@ -250,7 +252,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.get_ontology("name4") def test_i_can_access_values_after_push_and_pop_cache_only_true(self): - sheerka = self.get_sheerka(cache_only=True) + sheerka, context = self.init_test(cache_only=True).unpack() manager = SheerkaOntologyManager(sheerka, sheerka.root_folder, sheerka.cache_only) manager.register_cache("cache_name", Cache()) @@ -263,7 +265,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): manager.put("cache_name", "key", "value2") assert manager.get("cache_name", "key") == "value2" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value1" def test_i_can_access_values_after_push_and_pop_cache_only_false(self): @@ -296,7 +298,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[1].cache_manager.sdp.state.data == {'cache_name': {'key': 'value1'}} # remove a layer - manager.pop_ontology() + manager.pop_ontology(context) assert not manager.current_cache_manager().has("cache_name", "key") # value is no longer in cache assert manager.get("cache_name", "key") == "value1" @@ -333,13 +335,13 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.get("cache_name", "key") == "value4" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value3" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value2" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value1" def test_i_have_access_to_sub_layers_values_cache_only_false(self): @@ -368,6 +370,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): def test_i_can_get_value_from_all_layers(self): sheerka = self.get_sheerka(cache_only=False) + context = self.get_context(sheerka) manager = SheerkaOntologyManager(sheerka, sheerka.root_folder, sheerka.cache_only) manager.register_cache("cache_name", Cache().auto_configure("cache_name")) @@ -381,14 +384,15 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.get("cache_name", "key") == "value" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value" def test_i_can_only_get_top_layer_values_when_dictionary_cache(self): sheerka = self.get_sheerka() + context = self.get_context(sheerka) manager = SheerkaOntologyManager(sheerka, sheerka.root_folder, sheerka.cache_only) manager.register_cache("cache_name", DictionaryCache().auto_configure("cache_name")) @@ -407,7 +411,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.get_all("cache_name") == {"key": "value", "key1": "value1"} # I can get back my values after pop - manager.pop_ontology() + manager.pop_ontology(context) assert manager.copy("cache_name") == {"key": "value"} def test_dictionary_caches_values_are_copied_when_a_new_ontology_is_pushed(self): @@ -728,7 +732,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[1].cache_manager.sdp.state.data == {'cache_name': {"key": "value"}} # The entry still exists in lower ontology - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value" def test_i_can_remove_when_value_is_in_both_low_and_current_level(self): @@ -764,7 +768,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[1].cache_manager.sdp.state.data == {'cache_name': {"key": "value"}} # The entry still exists in lower ontology - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value" def test_i_can_remove_when_value_is_not_low_level(self): @@ -830,8 +834,8 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[2].cache_manager.sdp.state.data == {'cache_name': {"key": ["value", "value2"]}} # The entry still exists in lower ontology - manager.pop_ontology() - manager.pop_ontology() + manager.pop_ontology(context) + manager.pop_ontology(context) assert manager.get("cache_name", "key") == ["value", "value2"] def test_i_can_add_concept_default_layer(self): @@ -941,7 +945,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[1].cache_manager.sdp.get('by_id', foo.id) == foo # so I can get the old values when I pop ontology - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("by_key", foo.key) == foo assert manager.get("by_id", foo.id) == foo @@ -990,7 +994,7 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[1].cache_manager.sdp.get('by_id', foo.id) == foo # so I can get the old values when I pop ontology - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("by_key", foo.key) == foo assert manager.get("by_id", foo.id) == foo @@ -1119,12 +1123,12 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[2].cache_manager.sdp.get("by_key") == {foo.key: foo} # So I can pop - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("by_id", foo.id) == foo assert manager.get("by_key", foo.key) == foo # and pop again - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("by_id", foo.id) == foo assert manager.get("by_key", foo.key) == foo @@ -1167,6 +1171,145 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): "key5": "value5" } + def test_i_can_keep_track_of_created_concepts_by_ontologies(self): + sheerka, context, foo = self.init_concepts("foo", create_new=True) + + def from_cache(entry): + return sheerka.om.self_cache_manager.copy(entry) + + def from_db(entry): + return sheerka.om.self_cache_manager.sdp.get(entry) + + # check that the new concept is tracked + assert from_cache(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {'#unit_test#': {'1001'}} + assert from_cache(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {'1001': '#unit_test#'} + + # add a new ontology and make sure the new concepts are tracked + sheerka.push_ontology(context, "new ontology") + sheerka.create_new_concept(context, Concept("bar")) + assert from_cache(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {'#unit_test#': {'1001'}, + 'new ontology': {'1002'}} + assert from_cache(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {'1001': '#unit_test#', + '1002': 'new ontology'} + + # commit the info and check the DB + sheerka.om.commit(context) + assert from_db(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {'#unit_test#': {'1001'}, + 'new ontology': {'1002'}, } + assert from_db(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {'1001': '#unit_test#', + '1002': 'new ontology', } + + # remove a concept a check + sheerka.remove_concept(context, sheerka.get_by_name("foo")) + assert from_cache(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {'new ontology': {'1002'}, } + assert from_cache(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {'1002': 'new ontology', } + + sheerka.remove_concept(context, sheerka.get_by_name("bar")) + assert from_cache(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {} + assert from_cache(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {} + + # commit again and check + sheerka.om.commit(context) + assert from_db(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {} + assert from_db(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {} + + def test_i_can_keep_track_of_created_rules_by_ontologies(self): + sheerka, context, rule1 = self.init_format_rules(("rule1", "id.attr == 'value'", "True")) + + def rules_by_ontology_from_cache(): + res = sheerka.om.self_cache_manager.copy(SheerkaOntologyManager.RULES_BY_ONTOLOGY_ENTRY) + del res[SheerkaOntologyManager.ROOT_ONTOLOGY_NAME] # discard builtin rules + return res + + def ontologies_from_cache(): + res = sheerka.om.self_cache_manager.copy(SheerkaOntologyManager.ONTOLOGY_BY_RULE_ENTRY) + return {k: v for k, v in res.items() if v != SheerkaOntologyManager.ROOT_ONTOLOGY_NAME} + + def rules_by_ontology_from_db(): + res = sheerka.om.self_cache_manager.sdp.get(SheerkaOntologyManager.RULES_BY_ONTOLOGY_ENTRY) + del res[SheerkaOntologyManager.ROOT_ONTOLOGY_NAME] # discard builtin rules + return res + + def ontologies_from_db(): + res = sheerka.om.self_cache_manager.sdp.get(SheerkaOntologyManager.ONTOLOGY_BY_RULE_ENTRY) + return {k: v for k, v in res.items() if v != SheerkaOntologyManager.ROOT_ONTOLOGY_NAME} + + assert rules_by_ontology_from_cache() == {"#unit_test#": {rule1.id}} + assert ontologies_from_cache() == {rule1.id: "#unit_test#"} + + # add a new rule from a new ontology and check + sheerka.push_ontology(context, "new ontology") + rule2 = Rule(ACTION_TYPE_EXEC, "rule2", "id2.attr2 == 'value'", "True") + sheerka.create_new_rule(context, rule2) + + assert rules_by_ontology_from_cache() == {"#unit_test#": {rule1.id}, "new ontology": {rule2.id}} + assert ontologies_from_cache() == {rule1.id: "#unit_test#", rule2.id: "new ontology"} + + # commit and check the result + sheerka.om.commit(context) + assert rules_by_ontology_from_db() == {"#unit_test#": {rule1.id}, "new ontology": {rule2.id}} + assert ontologies_from_db() == {rule1.id: "#unit_test#", rule2.id: "new ontology"} + + sheerka.remove_rule(context, rule1) + assert rules_by_ontology_from_cache() == {"new ontology": {rule2.id}} + assert ontologies_from_cache() == {rule2.id: "new ontology"} + + # remove the last rule + sheerka.remove_rule(context, rule2) + assert rules_by_ontology_from_cache() == {} + assert ontologies_from_cache() == {} + + # commit and check the db + sheerka.om.commit(context) + assert rules_by_ontology_from_db() == {} + assert ontologies_from_db() == {} + + def test_i_can_keep_track_of_created_concept_on_ontology_pop(self): + sheerka, context, foo = self.init_concepts("foo", create_new=True) + + events_raised = set() + sheerka.subscribe(EVENT_CONCEPT_ID_DELETED, lambda ctx, c: events_raised.add(c)) + + def from_cache(entry): + return sheerka.om.self_cache_manager.copy(entry) + + sheerka.push_ontology(context, "new ontology") + sheerka.create_new_concept(context, Concept("bar")) + sheerka.create_new_concept(context, Concept("baz")) + + sheerka.pop_ontology(context) + assert from_cache(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == {'#unit_test#': {'1001'}} + assert from_cache(SheerkaOntologyManager.ONTOLOGY_BY_CONCEPT_ENTRY) == {'1001': '#unit_test#'} + + # check that the 'concept is deleted' events are raised + assert events_raised == {'1002', '1003'} + + def test_i_can_keep_track_of_created_rules_on_ontology_pop(self): + sheerka, context, rule1 = self.init_format_rules(("rule1", "id.attr == 'value'", "True")) + + events_raised = set() + sheerka.subscribe(EVENT_RULE_ID_DELETED, lambda ctx, r: events_raised.add(r)) + + def rules_by_ontology_from_cache(): + res = sheerka.om.self_cache_manager.copy(SheerkaOntologyManager.RULES_BY_ONTOLOGY_ENTRY) + del res[SheerkaOntologyManager.ROOT_ONTOLOGY_NAME] # discard builtin rules + return res + + def ontologies_from_cache(): + res = sheerka.om.self_cache_manager.copy(SheerkaOntologyManager.ONTOLOGY_BY_RULE_ENTRY) + return {k: v for k, v in res.items() if v != SheerkaOntologyManager.ROOT_ONTOLOGY_NAME} + + sheerka.push_ontology(context, "new ontology") + sheerka.create_new_rule(context, Rule(ACTION_TYPE_EXEC, "rule2", "id2.attr2 == 'value'", "True")) + sheerka.create_new_rule(context, Rule(ACTION_TYPE_EXEC, "rule3", "id3.attr3 == 'value'", "True")) + + sheerka.pop_ontology(context) + assert rules_by_ontology_from_cache() == {'#unit_test#': {'10'}} + assert ontologies_from_cache() == {'10': '#unit_test#'} + + # check that the 'rule is deleted' events are raised + assert events_raised == {'11', '12'} + # def test_i_can_list_by_key_when_dictionaries(self): # sheerka = self.get_sheerka(cache_only=False) # context = self.get_context(sheerka) @@ -1268,7 +1411,6 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): # assert manager.list_by_key("cache_name", "key2") == ["e", "f", "g"] # assert manager.list_by_key("cache_name", "key3") == ["a", "b", "c", "e", "f", "g"] - def test_i_can_get_call_when_a_cache_is_cleared(self): sheerka = self.get_sheerka(cache_only=False) context = self.get_context(sheerka) @@ -1401,11 +1543,11 @@ class TestSheerkaOntology(TestUsingMemoryBasedSheerka): assert manager.ontologies[2].cache_manager.get_cache("cache_name").copy() == {'key1': 'value1', 'key2': 'value2'} - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key1") == "new value1" assert manager.get("cache_name", "key2") is NotFound - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key1") == "value1" assert manager.get("cache_name", "key2") == "value2" @@ -1453,9 +1595,63 @@ class TestSheerkaOntologyWithFileBasedSheerka(TestUsingFileBasedSheerka): assert manager.get("cache_name", "key") == "value2" - manager.pop_ontology() + manager.pop_ontology(context) assert manager.get("cache_name", "key") == "value1" # put back the previous ontology manager.push_ontology("new ontology") assert manager.get("cache_name", "key") == "value2" + + def test_i_can_remember_concept_and_rules_by_ontology(self): + sheerka, context, foo, r1 = self.init_test().with_concepts( + "foo", + create_new=True + ).with_format_rules( + ("rule1", "__ret", "True"), + ).unpack() + sheerka.om.commit(context) + + sheerka = self.new_sheerka_instance(False) + context = self.get_context(sheerka) + + sheerka.create_new_concept(context, Concept("bar")) + r2 = sheerka.create_new_rule(context, Rule(ACTION_TYPE_EXEC, "rule2", "__ret.status", "True")).body.body + sheerka.om.commit(context) + + sheerka.push_ontology(context, "new ontology") + sheerka.create_new_concept(context, Concept("baz")) + sheerka.create_new_rule(context, Rule(ACTION_TYPE_EXEC, "rule3", "id3.attr3 == 'value'", "True")) + sheerka.om.commit(context) + + sheerka = self.new_sheerka_instance(False) + context = self.get_context(sheerka) + + sheerka.push_ontology(context, "another ontology") + sheerka.create_new_concept(context, Concept("qux")) + r4 = sheerka.create_new_rule(context, Rule(ACTION_TYPE_EXEC, "rule4", "id4.attr4", "True")).body.body + sheerka.remove_concept(context, foo) + sheerka.remove_rule(context, r2) + sheerka.om.commit(context) + + assert sheerka.om.self_cache_manager.copy(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == { + '#unit_test#': {'1002'}, + 'another ontology': {'1004'}, + } + + assert sheerka.om.self_cache_manager.copy(SheerkaOntologyManager.RULES_BY_ONTOLOGY_ENTRY) == { + '#unit_test#': {r1.id}, + 'another ontology': {r4.id}, + } + + # in db + assert sheerka.om.self_cache_manager.sdp.get(SheerkaOntologyManager.CONCEPTS_BY_ONTOLOGY_ENTRY) == { + '#unit_test#': {'1002'}, + 'another ontology': {'1004'}, + 'new ontology': {'1003'}} + + rules_from_db = sheerka.om.self_cache_manager.sdp.get(SheerkaOntologyManager.RULES_BY_ONTOLOGY_ENTRY) + del rules_from_db["__default__"] + assert rules_from_db == { + '#unit_test#': {'10'}, + 'another ontology': {'13'}, + 'new ontology': {'12'}} diff --git a/tests/core/test_sheerka_printer.py b/tests/core/test_sheerka_printer.py index cc32bc5..6dd632e 100644 --- a/tests/core/test_sheerka_printer.py +++ b/tests/core/test_sheerka_printer.py @@ -368,19 +368,19 @@ second : 'value d' sheerka.print(lst) captured = capsys.readouterr() assert captured.out == """(1001)foo a b -a : 'value a' -b : 'value b' id : '1001' name: 'foo a b' key : 'foo __var__0 __var__1' +a : 'value a' +b : 'value b' body: **NotInit** self: (1001)foo a b (1001)foo a b -a : 'value c' -b : 'value d' id : '1001' name: 'foo a b' key : 'foo __var__0 __var__1' +a : 'value c' +b : 'value d' body: **NotInit** self: (1001)foo a b """ @@ -404,6 +404,9 @@ self: (1001)foo a b sheerka.print(lst) captured = capsys.readouterr() assert captured.out == """(1001)foo a b +id : '1001' +name: 'foo a b' +key : 'foo __var__0 __var__1' a : 'value a' b : {'a' : 'value a' 'beta' : {'b1': 10 @@ -425,9 +428,6 @@ b : {'a' : 'value a' 'empty': ()} 'h' : {'set' : {'set-a'} 'empty': {}}} -id : '1001' -name: 'foo a b' -key : 'foo __var__0 __var__1' body: **NotInit** self: (1001)foo a b """ diff --git a/tests/core/test_utils.py b/tests/core/test_utils.py index f59f318..4699cb5 100644 --- a/tests/core/test_utils.py +++ b/tests/core/test_utils.py @@ -1,7 +1,8 @@ from dataclasses import dataclass -import core.utils import pytest + +import core.utils from core.builtin_concepts import BuiltinConcepts from core.builtin_helpers import evaluate_expression from core.concept import Concept @@ -111,7 +112,7 @@ def test_i_can_get_sub_classes(): ([['a'], ['b']], ['c', 'd', 'e'], [['a', 'c'], ['b', 'c'], ['a', 'd'], ['b', 'd'], ['a', 'e'], ['b', 'e']]), ]) def test_i_can_product(a, b, expected): - res = core.utils.product(a, b) + res = core.utils.sheerka_product(a, b) assert res == expected @@ -425,3 +426,31 @@ def test_i_can_deep_copy_a_custom_type(): assert core.utils.sheerka_deepcopy(NotInit) is NotInit assert core.utils.sheerka_deepcopy(NotFound) is NotFound assert core.utils.sheerka_deepcopy(Removed) is Removed + + +@pytest.mark.parametrize("expression1, expression2, expected", [ + ("foo bar baz", "foo bar baz", True), + ("foo()", " foo ( ) ", True), + ("is_instance()", "is _ instance()", False), + ("foo bar baz", "foo bar", False) +]) +def test_tokens_are_matching(expression1, expression2, expected): + assert core.utils.tokens_are_matching(Tokenizer(expression1), Tokenizer(expression2)) == expected + + +def test_tokens_are_matching_when_no_eof(): + expression1 = "foo bar baz" + expression2 = "foo bar" + tokens1 = Tokenizer(expression1, yield_eof=False) + tokens2 = Tokenizer(expression2, yield_eof=False) + + assert not core.utils.tokens_are_matching(tokens1, tokens2) + + +def test_tokens_are_matching_when_eof_differs(): + expression1 = "foo bar baz" + expression2 = "foo bar baz" + tokens1 = Tokenizer(expression1, yield_eof=True) + tokens2 = Tokenizer(expression2, yield_eof=False) + + assert core.utils.tokens_are_matching(tokens1, tokens2) diff --git a/tests/evaluators/test_DefRuleEvaluator.py b/tests/evaluators/test_DefRuleEvaluator.py new file mode 100644 index 0000000..4f7e29c --- /dev/null +++ b/tests/evaluators/test_DefRuleEvaluator.py @@ -0,0 +1,60 @@ +import pytest + + +from core.builtin_concepts import ReturnValueConcept, ParserResultConcept, BuiltinConcepts +from core.rule import Rule, RuleMetadata +from core.sheerka.services.SheerkaExecute import ParserInput +from core.tokenizer import Tokenizer +from evaluators.DefRuleEvaluator import DefRuleEvaluator +from parsers.DefRuleParser import DefFormatRuleNode, DefRuleParser, DefExecRuleNode +from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka + + +class TestDefRuleEvaluator(TestUsingMemoryBasedSheerka): + + @staticmethod + def get_ret_val_from_rule(rule_def): + tokens = {k: list(Tokenizer(k.value + " " + v, yield_eof=False)) for k, v in rule_def.items()} + for v in tokens.values(): + del v[1] + + node = DefFormatRuleNode(tokens) + return ReturnValueConcept("parsers.FormatRule", True, ParserResultConcept(value=node)) + + @pytest.mark.parametrize("ret_val, expected", [ + (ReturnValueConcept("name", True, ParserResultConcept(value=DefFormatRuleNode({}))), True), + (ReturnValueConcept("name", True, ParserResultConcept(value=DefExecRuleNode({}))), True), + (ReturnValueConcept("name", True, ParserResultConcept(value="other object")), False), + (ReturnValueConcept("name", False, ParserResultConcept(value=DefFormatRuleNode({}))), False), + (ReturnValueConcept("name", False, DefFormatRuleNode({})), False), + ]) + def test_i_can_match(self, ret_val, expected): + context = self.get_context() + assert DefRuleEvaluator().matches(context, ret_val) == expected + + @pytest.mark.parametrize("text, expected_action_type, expected_name", [ + ("when True print 'hello world'", "print", None), + ("when True then 'hello world'", "exec", None), + ("def rule rule name as when True print 'hello world'", "print", "rule name"), + ("def rule rule name as when True then 'hello world'", "exec", "rule name"), + ]) + def test_i_can_eval(self, text, expected_action_type, expected_name): + sheerka, context = self.init_concepts() + ret_val = DefRuleParser().parse(context, ParserInput(text)) + + res = DefRuleEvaluator().eval(context, ret_val) + + assert res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.NEW_RULE) + assert isinstance(res.body.body, Rule) + assert res.body.body.metadata == RuleMetadata(expected_action_type, + expected_name, + "True", + "'hello world'", + id=res.body.body.metadata.id, # no need to compare the id + is_compiled=True, + is_enabled=True) + rule = res.body.body + assert rule.compiled_predicates is not None + assert rule.compiled_action is not None + assert rule.rete_disjunctions is not None diff --git a/tests/evaluators/test_FormatRuleEvaluator.py b/tests/evaluators/test_FormatRuleEvaluator.py deleted file mode 100644 index 6a2c4d6..0000000 --- a/tests/evaluators/test_FormatRuleEvaluator.py +++ /dev/null @@ -1,51 +0,0 @@ -import pytest -from core.builtin_concepts import ReturnValueConcept, ParserResultConcept, BuiltinConcepts -from core.rule import Rule, RuleMetadata -from core.sheerka.services.SheerkaExecute import ParserInput -from core.tokenizer import Tokenizer -from evaluators.FormatRuleEvaluator import FormatRuleEvaluator -from parsers.DefFormatRuleParser import FormatRuleNode, DefFormatRuleParser - -from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka - - -class TestFormatRuleEvaluator(TestUsingMemoryBasedSheerka): - - @staticmethod - def get_ret_val_from_rule(rule_def): - tokens = {k: list(Tokenizer(k.value + " " + v, yield_eof=False)) for k, v in rule_def.items()} - for v in tokens.values(): - del v[1] - - node = FormatRuleNode(tokens) - return ReturnValueConcept("parsers.FormatRule", True, ParserResultConcept(value=node)) - - @pytest.mark.parametrize("ret_val, expected", [ - (ReturnValueConcept("name", True, ParserResultConcept(value=FormatRuleNode({}))), True), - (ReturnValueConcept("name", True, ParserResultConcept(value="other object")), False), - (ReturnValueConcept("name", False, ParserResultConcept(value=FormatRuleNode({}))), False), - (ReturnValueConcept("name", False, FormatRuleNode({})), False), - ]) - def test_i_can_match(self, ret_val, expected): - context = self.get_context() - assert FormatRuleEvaluator().matches(context, ret_val) == expected - - def test_i_can_eval(self): - sheerka, context = self.init_concepts() - text = "when isinstance(value, __EXPLANATION) print list(value)" - ret_val = DefFormatRuleParser().parse(context, ParserInput(text)) - - res = FormatRuleEvaluator().eval(context, ret_val) - - assert res.status - assert sheerka.isinstance(res.body, BuiltinConcepts.NEW_RULE) - assert isinstance(res.body.body, Rule) - assert res.body.body.metadata == RuleMetadata("print", - None, - "isinstance(value, __EXPLANATION)", - "list(value)", - id=res.body.body.metadata.id, # no need to compare the id - is_compiled=True, - is_enabled=True) - assert res.body.body.compiled_predicate is not None - assert res.body.body.compiled_action is not None diff --git a/tests/evaluators/test_LexerNodeEvaluator.py b/tests/evaluators/test_LexerNodeEvaluator.py index b26be69..a0941a1 100644 --- a/tests/evaluators/test_LexerNodeEvaluator.py +++ b/tests/evaluators/test_LexerNodeEvaluator.py @@ -87,6 +87,6 @@ class TestLexerNodeEvaluator(TestUsingMemoryBasedSheerka): assert context.sheerka.isinstance(wrapper, BuiltinConcepts.PARSER_RESULT) assert wrapper.source == "foo + 1" - assert return_value == PythonNode('foo + 1', ast.parse("__C__foo__C__ + 1", mode="eval")) + assert return_value == PythonNode('__C__foo__C__ + 1', ast.parse("__C__foo__C__ + 1", mode="eval"), "foo + 1") assert return_value.objects == {"__C__foo__C__": foo} assert result.parents == [ret_val] diff --git a/tests/evaluators/test_PythonEvaluator.py b/tests/evaluators/test_PythonEvaluator.py index 4a915cf..5558696 100644 --- a/tests/evaluators/test_PythonEvaluator.py +++ b/tests/evaluators/test_PythonEvaluator.py @@ -3,6 +3,7 @@ import ast import pytest from core.builtin_concepts import ReturnValueConcept, ParserResultConcept, BuiltinConcepts +from core.builtin_helpers import CreateObjectIdentifiers from core.concept import Concept, CB from core.sheerka.services.SheerkaConceptManager import SheerkaConceptManager from core.sheerka.services.SheerkaExecute import ParserInput @@ -26,7 +27,7 @@ def return_return_value(status): def get_source_code_node(source_code, concepts=None): if concepts: for concept_name, concept in sorted(concepts.items(), key=lambda kv: len(kv[0]), reverse=True): - identifier = "__C__" + PythonWithConceptsParser.sanitize(concept.name) + identifier = "__C__" + CreateObjectIdentifiers.sanitize(concept.name) if concept.id: identifier += "__" + concept.id identifier += "__C__" diff --git a/tests/non_reg/test_sheerka_non_reg.py b/tests/non_reg/test_sheerka_non_reg.py index e992107..953aec9 100644 --- a/tests/non_reg/test_sheerka_non_reg.py +++ b/tests/non_reg/test_sheerka_non_reg.py @@ -1230,7 +1230,7 @@ as: assert res[0].body == 21 def test_i_can_define_rules_priorities(self): - sheerka, context, r1, r2 = self.init_test().with_rules(("True", "True"), ("False", "False")).unpack() + sheerka, context, r1, r2 = self.init_test().with_format_rules(("True", "True"), ("False", "False")).unpack() sheerka.evaluate_user_input("def concept rule x > rule y where isinstance(x, int) and isinstance(y, int) as set_is_greater_than(__PRECEDENCE, r:|x:, r:|y:, 'Rule')") res = sheerka.evaluate_user_input(f"eval rule {r1.id} > rule {r2.id}") diff --git a/tests/non_reg/test_sheerka_display.py b/tests/non_reg/test_sheerka_non_reg_out.py similarity index 87% rename from tests/non_reg/test_sheerka_display.py rename to tests/non_reg/test_sheerka_non_reg_out.py index ac03825..bf1cec6 100644 --- a/tests/non_reg/test_sheerka_display.py +++ b/tests/non_reg/test_sheerka_non_reg_out.py @@ -51,6 +51,23 @@ post : None ret : None vars : [] props : {} +""" + + def test_i_can_describe_a_rule(self, capsys): + sheerka = self.get_sheerka() + + sheerka.enable_process_return_values = True + sheerka.evaluate_user_input("desc(r:|1:)") + captured = capsys.readouterr() + + assert captured.out == """id : 1 +name : Print return values +type : print +predicate: __rets +action : list(__rets) +priority : 4 +compiled : True +enabled : True """ def test_i_can_display_multiple_success(self, capsys): diff --git a/tests/non_reg/test_sheerka_non_reg_rules.py b/tests/non_reg/test_sheerka_non_reg_rules.py new file mode 100644 index 0000000..78b037e --- /dev/null +++ b/tests/non_reg/test_sheerka_non_reg_rules.py @@ -0,0 +1,25 @@ +from core.builtin_concepts_ids import BuiltinConcepts +from evaluators.PythonEvaluator import PythonEvalError +from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka + + +class TestSheerkaNonRegDisplay(TestUsingMemoryBasedSheerka): + + def test_i_can_apply_simple_rule(self): + init = [ + "def concept one as 1", + "when __ret.status then test_error()", + ] + + sheerka = self.init_scenario(init) + sheerka.enable_process_rules = True + res = sheerka.evaluate_user_input("one") + + assert len(res) == 1 + ret = res[0] + + assert not ret.status + assert sheerka.isinstance(ret.body, BuiltinConcepts.ERROR) + assert isinstance(ret.body.body, PythonEvalError) + assert isinstance(ret.body.body.error, Exception) + assert ret.body.body.error.args == ("I can raise an error",) diff --git a/tests/out/test_SheerkaOut.py b/tests/out/test_SheerkaOut.py index c8276a7..969a58c 100644 --- a/tests/out/test_SheerkaOut.py +++ b/tests/out/test_SheerkaOut.py @@ -38,7 +38,7 @@ class TestSheerkaOut(TestUsingMemoryBasedSheerka): @classmethod def teardown_class(cls): - cls.sheerka.pop_ontology() + cls.sheerka.pop_ontology(TestSheerkaOut.context) cls.root_ontology_name = SheerkaOntologyManager.ROOT_ONTOLOGY_NAME def init_service_with_rules(self, *rules, **kwargs): diff --git a/tests/parsers/parsers_utils.py b/tests/parsers/parsers_utils.py index fbcbf9a..c97c8d9 100644 --- a/tests/parsers/parsers_utils.py +++ b/tests/parsers/parsers_utils.py @@ -1,8 +1,193 @@ -from core.concept import CC, Concept, ConceptParts, DoNotResolve, CIO +import ast +from dataclasses import dataclass + +from core.builtin_helpers import CreateObjectIdentifiers +from core.concept import CC, Concept, ConceptParts, DoNotResolve, CIO, CMV from core.tokenizer import Tokenizer, TokenKind, Token -from parsers.BaseNodeParser import scnode, utnode, cnode, SCWC, CNC, short_cnode, SourceCodeWithConceptNode, CN, UTN, \ - SCN, RN +from core.utils import get_text_from_tokens, tokens_index +from parsers.BaseNodeParser import scnode, utnode, cnode, SCWC, CNC, short_cnode, CN, UTN, \ + SCN, RN, UnrecognizedTokensNode, SourceCodeNode +from parsers.PythonParser import PythonNode from parsers.SyaNodeParser import SyaConceptParserHelper +from parsers.expressions import NameExprNode, AndNode, OrNode, NotNode, VariableNode, ComparisonNode, ComparisonType +from sheerkarete.common import V +from sheerkarete.conditions import Condition, AndConditions + + +@dataclass +class Obj: + prop_a: object + prop_b: object = None + prop_c: object = None + parent: object = None + + +class AND: + """ Test class for AndNode""" + + def __init__(self, *parts, source=None): + self.parts = parts + self.source = source + + +class OR: + """ Test class for OrNode""" + + def __init__(self, *parts, source=None): + self.parts = parts + self.source = source + + +@dataclass +class NOT: + """ Test class for NotNode""" + expr: object + source: str = None + + +@dataclass +class EXPR: + """Test class for NameNode. E stands for Expression""" + source: str + + +@dataclass +class VAR: + """Test class for VarNode""" + full_name: str + source: str = None + + +@dataclass +class EQ: + left: object + right: object + source = None + + +@dataclass +class NEQ: + left: object + right: object + source = None + + +@dataclass +class GT: + left: object + right: object + source = None + + +@dataclass +class GTE: + left: object + right: object + source = None + + +@dataclass +class LT: + left: object + right: object + source = None + + +@dataclass +class LTE: + left: object + right: object + source = None + + +@dataclass +class IN: + left: object + right: object + source = None + + +@dataclass +class NIN: # for NOT INT + left: object + right: object + source = None + + +comparison_type_mapping = { + "EQ": ComparisonType.EQUALS, + "NEQ": ComparisonType.NOT_EQUAlS, + "LT": ComparisonType.LESS_THAN, + "LTE": ComparisonType.LESS_THAN_OR_EQUALS, + "GT": ComparisonType.GREATER_THAN, + "GTE": ComparisonType.GREATER_THAN_OR_EQUALS, + "IN": ComparisonType.IN, + "NIN": ComparisonType.NOT_IN, +} + + +def get_expr_node_from_test_node(full_text, test_node): + """ + Returns EXPR, OR, NOT, AND object to ease the comparison with the real ExprNode + """ + full_text_as_tokens = list(Tokenizer(full_text, yield_eof=False)) + + def get_pos(nodes): + start, end = None, None + for n in nodes: + if start is None or start > n.start: + start = n.start + if end is None or end < n.end: + end = n.end + return start, end + + def get_pos_from_source(source): + source_as_node = list(Tokenizer(source, yield_eof=False)) + start = tokens_index(full_text_as_tokens, source_as_node) + end = start + len(source_as_node) - 1 + return start, end + + def get_expr_node(node): + + if isinstance(node, EXPR): + value_as_tokens = list(Tokenizer(node.source, yield_eof=False)) + start = tokens_index(full_text_as_tokens, value_as_tokens, 0) + end = start + len(value_as_tokens) - 1 + return NameExprNode(start, end, full_text_as_tokens[start: end + 1]) + + if isinstance(node, AND): + parts = [get_expr_node(part) for part in node.parts] + start, end = get_pos_from_source(node.source) if node.source else get_pos(parts) + return AndNode(start, end, full_text_as_tokens[start: end + 1], *parts) + + if isinstance(node, OR): + parts = [get_expr_node(part) for part in node.parts] + start, end = get_pos_from_source(node.source) if node.source else get_pos(parts) + return OrNode(start, end, full_text_as_tokens[start: end + 1], *parts) + + if isinstance(node, NOT): + part = get_expr_node(node.expr) + start, end = get_pos_from_source(node.source) if node.source else (part.start - 2, part.end) + return NotNode(start, end, full_text_as_tokens[start: end + 1], part) + + if isinstance(node, VAR): + value_as_tokens = list(Tokenizer(node.source or node.full_name, yield_eof=False)) + start = tokens_index(full_text_as_tokens, value_as_tokens, 0) + end = start + len(value_as_tokens) - 1 + parts = node.full_name.split(".") + if len(parts) == 1: + return VariableNode(start, end, full_text_as_tokens[start: end + 1], parts[0]) + else: + return VariableNode(start, end, full_text_as_tokens[start: end + 1], parts[0], *parts[1:]) + + if isinstance(node, (EQ, NEQ, GT, GTE, LT, LTE, IN, NIN)): + node_type = comparison_type_mapping[type(node).__name__] + left_node, right_node = get_expr_node(node.left), get_expr_node(node.right) + start, end = get_pos_from_source(node.source) if node.source else get_pos([left_node, right_node]) + return ComparisonNode(start, end, full_text_as_tokens[start: end + 1], + node_type, left_node, right_node) + + return get_expr_node(test_node) def _index(tokens, expr, index): @@ -101,7 +286,7 @@ def get_node( sub_expr.fix_pos(sub_expr.first) sub_expr.fix_pos(sub_expr.last) return sub_expr - #return SourceCodeWithConceptNode(first, last, content).pseudo_fix_source() + # return SourceCodeWithConceptNode(first, last, content).pseudo_fix_source() if isinstance(sub_expr, SCN): node = get_node(concepts_map, expression_as_tokens, sub_expr.source, sya=sya) @@ -128,7 +313,8 @@ def get_node( sub_expr.fix_pos((concept_node.start, concept_node.end if hasattr(concept_node, "end") else concept_node.start)) if hasattr(sub_expr, "compiled"): for k, v in sub_expr.compiled.items(): - node = get_node(concepts_map, expression_as_tokens, v, sya=sya, exclude_body=exclude_body) # need to get start and end positions + node = get_node(concepts_map, expression_as_tokens, v, sya=sya, + exclude_body=exclude_body) # need to get start and end positions if isinstance(v, str) and v in concepts_map: new_value_concept = concepts_map[v] new_value = CC(Concept().update_from(new_value_concept), exclude_body=exclude_body) @@ -214,3 +400,85 @@ def compute_expected_array(concepts_map, expression, expected, sya=False, init_e sya=sya, init_empty_body=init_empty_body, exclude_body=exclude_body) for sub_expr in expected] + + +def get_unrecognized_node(start, text): + tokens = list(Tokenizer(text, yield_eof=False)) + return UnrecognizedTokensNode(start, start + len(tokens) - 1, tokens) + + +def get_source_code_node(start, text, concepts_map, id_manager=None): + id_manager = id_manager or CreateObjectIdentifiers() + id_mapping = {} + concept_mapping_by_id = {} + + # get the concepts, mapped by their new id + for concept_name, concept in concepts_map.items(): + concept_identifier = id_manager.get_identifier(concept, "__C__") + id_mapping[concept_name] = concept_identifier + concept_mapping_by_id[concept_identifier] = concept + + # transform the source code to use the new id + tokens = list(Tokenizer(text, yield_eof=False)) + text_to_compile_tokens = [] + for t in tokens: + if t.type == TokenKind.IDENTIFIER and t.value in id_mapping: + text_to_compile_tokens.append(Token(TokenKind.IDENTIFIER, id_mapping[t.value], -1, -1, -1)) + else: + text_to_compile_tokens.append(t) + text_to_compile = get_text_from_tokens(text_to_compile_tokens) + + # create the python node + ast_ = ast.parse(text_to_compile, "", 'eval') + python_node = PythonNode(text_to_compile, ast_, text) + python_node.objects = concept_mapping_by_id + + return SourceCodeNode(start, start + len(tokens) - 1, tokens, text, python_node) + + +def resolve_test_concept(concept_map, hint): + if isinstance(hint, str): + return concept_map[hint] + + if isinstance(hint, CC): + concept = concept_map[hint.concept_key] + compiled = {k: resolve_test_concept(concept_map, v) for k, v in hint.compiled.items()} + return CC(concept, source=hint.source, exclude_body=hint.exclude_body, **compiled) + + if isinstance(hint, CMV): + concept = concept_map[hint.concept_key] + return CMV(concept, **hint.variables) + + # CV + # + # CMV + # + # CIO + raise NotImplementedError() + + +def get_rete_conditions(*conditions_as_string): + """ + Transform a list of string into a list of Condition (Rete conditions) + :param conditions_as_string: conditions in the form 'identifier|attribute|value' + when one argument starts with "#" it means that it's a variables + ex : "#__x_00__|__name__|'__ret'" -> Condition(V('#__x_00__'), '__name__', '__ret') + + Caution, the value part is evaluated + "identifier|__name__|'True'" -> Condition(identifier, '__name__', 'True') # the string 'True' + "identifier|__name__|True" -> Condition(identifier, '__name__', True) # the bool True + """ + res = [] + for as_string in conditions_as_string: + identifier, attribute, value = as_string.split("|") + if identifier.startswith("#"): + identifier = V(identifier[1:]) + if value.startswith("'"): + value = value[1:-1] + elif value in ("True", "False"): + value = (value == "True") + else: + value = int(value) + + res.append(Condition(identifier, attribute, value)) + return AndConditions(res) diff --git a/tests/parsers/test_BaseCustomGrammarParser.py b/tests/parsers/test_BaseCustomGrammarParser.py index ce0ef9c..e6022cd 100644 --- a/tests/parsers/test_BaseCustomGrammarParser.py +++ b/tests/parsers/test_BaseCustomGrammarParser.py @@ -1,9 +1,9 @@ import pytest + from core.sheerka.services.SheerkaExecute import ParserInput from core.tokenizer import Keywords, Tokenizer, TokenKind from parsers.BaseCustomGrammarParser import BaseCustomGrammarParser, SyntaxErrorNode, KeywordNotFound from parsers.BaseParser import UnexpectedEofParsingError, UnexpectedTokenParsingError - from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka @@ -43,6 +43,7 @@ class TestBaseCustomGrammarParser(TestUsingMemoryBasedSheerka): ("when uuu vvv print xxx yyy", False, {Keywords.WHEN: "when uuu vvv ", Keywords.PRINT: "print xxx yyy"}), ("print xxx yyy when uuu vvv", False, {Keywords.WHEN: "when uuu vvv", Keywords.PRINT: "print xxx yyy "}), (" when xxx", False, {Keywords.WHEN: "when xxx"}), + ("when xxx yyy", True, {Keywords.WHEN: "when xxx yyy"}), ("when uuu vvv print xxx yyy", True, {Keywords.WHEN: "when uuu vvv", Keywords.PRINT: "print xxx yyy"}), ("print xxx yyy when uuu vvv", True, {Keywords.WHEN: "when uuu vvv", Keywords.PRINT: "print xxx yyy"}), @@ -84,7 +85,7 @@ func(a) assert parser.get_parts(["when", "print"], Keywords.PRINT) is None assert parser.error_sink == [UnexpectedTokenParsingError(f"'print' keyword not found.", - "when", + "when", [Keywords.PRINT])] def test_i_can_detect_when_a_keyword_appears_several_times(self): @@ -214,7 +215,8 @@ print xxx""" sheerka, context, parser = self.init_parser("") assert parser.get_body(list(Tokenizer(text, yield_eof=False))) is None - assert parser.error_sink == [UnexpectedTokenParsingError("Indentation not found.", "zzz", [TokenKind.WHITESPACE])] + assert parser.error_sink == [ + UnexpectedTokenParsingError("Indentation not found.", "zzz", [TokenKind.WHITESPACE])] def test_i_can_detect_invalid_indentation_when_get_body(self): sheerka, context, parser = self.init_parser("") diff --git a/tests/parsers/test_BnfNodeParser.py b/tests/parsers/test_BnfNodeParser.py index b958192..3733fab 100644 --- a/tests/parsers/test_BnfNodeParser.py +++ b/tests/parsers/test_BnfNodeParser.py @@ -136,7 +136,7 @@ class TestBnfNodeParser(TestUsingMemoryBasedSheerka): sheerka.set_isa(context, sheerka.new("thousands"), sheerka.new("number")) cls.shared_ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() + sheerka.pop_ontology(context) @staticmethod def update_bnf(context, concept): diff --git a/tests/parsers/test_ComparisonParser.py b/tests/parsers/test_ComparisonParser.py new file mode 100644 index 0000000..6648a35 --- /dev/null +++ b/tests/parsers/test_ComparisonParser.py @@ -0,0 +1,98 @@ +import pytest + +from core.builtin_concepts_ids import BuiltinConcepts +from core.sheerka.services.SheerkaExecute import ParserInput +from core.tokenizer import TokenKind, Tokenizer +from parsers.ComparisonParser import ComparisonParser +from parsers.ExpressionParser import ParenthesisMismatchError +from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka +from tests.parsers.parsers_utils import get_expr_node_from_test_node, VAR, EXPR, EQ, NEQ, GT, GTE, LT, LTE, IN, NIN + + +class TestComparisonParser(TestUsingMemoryBasedSheerka): + def init_parser(self): + sheerka, context = self.init_concepts() + parser = ComparisonParser() + return sheerka, context, parser + + def test_i_can_detect_empty_expression(self): + sheerka, context, parser = self.init_parser() + res = parser.parse(context, ParserInput("")) + + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.IS_EMPTY) + + @pytest.mark.parametrize("expression, expected", [ + ("var_name", VAR("var_name")), + ("var_name.attr", VAR("var_name.attr")), + ("var_name .attr", VAR("var_name.attr", source="var_name .attr")), + ("var_name. attr", VAR("var_name.attr", source="var_name. attr")), + ("var_name . attr", VAR("var_name.attr", source="var_name . attr")), + ("var_name.attr.get_value(x)", VAR("var_name.attr.get_value(x)")), + ("var_name.attr == 10", EQ(VAR("var_name.attr"), EXPR("10"))), + ("var_name.attr != 10", NEQ(VAR("var_name.attr"), EXPR("10"))), + ("var_name.attr > 10", GT(VAR("var_name.attr"), EXPR("10"))), + ("var_name.attr >= 10", GTE(VAR("var_name.attr"), EXPR("10"))), + ("var_name.attr < 10", LT(VAR("var_name.attr"), EXPR("10"))), + ("var_name.attr <= 10", LTE(VAR("var_name.attr"), EXPR("10"))), + ("var_name.attr in (a, b)", IN(VAR("var_name.attr"), EXPR("(a, b)"))), + ("var_name.attr not in (a, b)", NIN(VAR("var_name.attr"), EXPR("(a, b)"))), + ("var1.attr1 == var2.attr2", EQ(VAR("var1.attr1"), VAR("var2.attr2"))), + + ("not a var identifier", EXPR("not a var identifier")), + ("func()", EXPR("func()")), + ("func(a, not an identifier, x >5)", EXPR("func(a, not an identifier, x >5)")), + ]) + def test_i_can_parse_simple_expressions(self, expression, expected): + sheerka, context, parser = self.init_parser() + expected = get_expr_node_from_test_node(expression, expected) + + res = parser.parse(context, ParserInput(expression)) + parser_result = res.body + parsed_expr = parser_result.body + + assert res.status + assert res.who == parser.name + assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) + assert parsed_expr == expected + + @pytest.mark.parametrize("expression, expected_error, parenthesis_type, index", [ + ("(", BuiltinConcepts.NOT_FOR_ME, TokenKind.LPAR, 0), + (")", BuiltinConcepts.NOT_FOR_ME, TokenKind.RPAR, 0), + ("something (", BuiltinConcepts.NOT_FOR_ME, TokenKind.LPAR, 10), + ("something )", BuiltinConcepts.NOT_FOR_ME, TokenKind.RPAR, 10), + ("something == (", BuiltinConcepts.ERROR, TokenKind.LPAR, 13), + ("something == )", BuiltinConcepts.ERROR, TokenKind.RPAR, 13), + ("something (==", BuiltinConcepts.NOT_FOR_ME, TokenKind.LPAR, 10), + ("something )==", BuiltinConcepts.NOT_FOR_ME, TokenKind.RPAR, 10), + ]) + def test_i_can_detect_unbalanced_parenthesis(self, expression, expected_error, parenthesis_type, index): + sheerka, context, parser = self.init_parser() + + res = parser.parse(context, ParserInput(expression)) + assert not res.status + if expected_error == BuiltinConcepts.NOT_FOR_ME: + assert sheerka.isinstance(res.body, BuiltinConcepts.NOT_FOR_ME) + assert isinstance(res.body.reason[0], ParenthesisMismatchError) + assert res.body.reason[0].token.type == parenthesis_type + assert res.body.reason[0].token.index == index + else: + assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) + assert isinstance(res.body.body[0], ParenthesisMismatchError) + assert res.body.body[0].token.type == parenthesis_type + assert res.body.body[0].token.index == index + + def test_i_can_parse_tokens_rather_than_parser_input(self): + sheerka, context, parser = self.init_parser() + expression = "var1.attr1 == var2.attr2" + expected = EQ(VAR("var1.attr1"), VAR("var2.attr2")) + expected = get_expr_node_from_test_node(expression, expected) + + res = parser.parse(context, list(Tokenizer(expression))) + parser_result = res.body + parsed_expr = parser_result.body + + assert res.status + assert res.who == parser.name + assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) + assert parsed_expr == expected diff --git a/tests/parsers/test_DefConceptParser.py b/tests/parsers/test_DefConceptParser.py index caf1ce8..2266db6 100644 --- a/tests/parsers/test_DefConceptParser.py +++ b/tests/parsers/test_DefConceptParser.py @@ -4,10 +4,11 @@ from dataclasses import dataclass import pytest from core.builtin_concepts import ParserResultConcept, BuiltinConcepts, ReturnValueConcept from core.concept import DEFINITION_TYPE_BNF, DEFINITION_TYPE_DEF, Concept, CV +from core.global_symbols import NotInit from core.sheerka.services.SheerkaExecute import ParserInput from core.tokenizer import Keywords, Tokenizer, LexerError from parsers.BaseNodeParser import SCWC -from parsers.BaseParser import NotInitializedNode, UnexpectedEofParsingError +from parsers.BaseParser import UnexpectedEofParsingError from parsers.BnfNodeParser import OrderedChoice, ConceptExpression, StrMatch, Sequence from parsers.BnfDefinitionParser import BnfDefinitionParser from parsers.DefConceptParser import DefConceptParser, NameNode, SyntaxErrorNode @@ -222,11 +223,11 @@ class TestDefConceptParser(TestUsingMemoryBasedSheerka): assert res.status assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) assert isinstance(node, DefConceptNode) - assert node.body == NotInitializedNode() - assert node.where == NotInitializedNode() - assert node.pre == NotInitializedNode() - assert node.post == NotInitializedNode() - assert node.ret == NotInitializedNode() + assert node.body is NotInit + assert node.where is NotInit + assert node.pre is NotInit + assert node.post is NotInit + assert node.ret is NotInit @pytest.mark.parametrize("part", [ "as", diff --git a/tests/parsers/test_DefFormatRuleParser.py b/tests/parsers/test_DefFormatRuleParser.py deleted file mode 100644 index 913fc2f..0000000 --- a/tests/parsers/test_DefFormatRuleParser.py +++ /dev/null @@ -1,143 +0,0 @@ -import pytest -from core.builtin_concepts import BuiltinConcepts -from core.concept import Concept -from core.sheerka.services.SheerkaExecute import ParserInput -from core.sheerka.services.SheerkaRuleManager import FormatAstRawText, RulePredicate, FormatAstVariable -from core.tokenizer import Keywords, Tokenizer -from parsers.BaseCustomGrammarParser import KeywordNotFound -from parsers.DefFormatRuleParser import DefFormatRuleParser, FormatRuleNode - -from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka - -cmap = { - "is a": Concept("x is a y").def_var("x").def_var("y"), - "is a question": Concept("x is a y", pre="is_question()").def_var("x").def_var("y"), - "a is good": Concept("a is good").def_var("a"), - "b is good": Concept("b is good").def_var("b"), -} - - -class TestDefFormatRuleParser(TestUsingMemoryBasedSheerka): - shared_ontology = None - - @classmethod - def setup_class(cls): - init_test_helper = cls().init_test(cache_only=False, ontology="#TestDefFormatRuleParser#") - sheerka, context, *updated = init_test_helper.with_concepts(*cmap.values(), create_new=True).unpack() - for i, concept_name in enumerate(cmap): - cmap[concept_name] = updated[i] - - cls.shared_ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() - - def init_parser(self, my_concepts_map=None, **kwargs): - if my_concepts_map is None: - sheerka, context = self.init_test().unpack() - sheerka.add_ontology(context, self.shared_ontology) - else: - sheerka, context, *updated = self.init_test().with_concepts(*my_concepts_map.values(), **kwargs).unpack() - for i, pair in enumerate(my_concepts_map): - my_concepts_map[pair] = updated[i] - - parser = DefFormatRuleParser() - return sheerka, context, parser - - def test_i_can_detect_empty_expression(self): - sheerka, context, parser = self.init_parser() - res = parser.parse(context, ParserInput("")) - - assert not res.status - assert sheerka.isinstance(res.body, BuiltinConcepts.IS_EMPTY) - - def test_input_must_be_a_parser_input(self): - sheerka, context, parser = self.init_parser() - parser.parse(context, "not a parser input") is None - - def test_i_can_parse_a_simple_rule(self): - sheerka, context, parser = self.init_parser() - - text = "when isinstance(last_value(), Concept) print hello world!" - res = parser.parse(context, ParserInput(text)) - parser_result = res.body - format_rule = res.body.body - rules = format_rule.rule - format_ast = format_rule.format_ast - - assert res.status - assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) - assert isinstance(format_rule, FormatRuleNode) - assert self.dump_tokens(format_rule.tokens[Keywords.WHEN][1:]) == self.dump_tokens( - Tokenizer("isinstance(last_value(), Concept)", False)) - assert self.dump_tokens(format_rule.tokens[Keywords.PRINT][1:]) == self.dump_tokens( - Tokenizer("hello world!", False)) - - assert len(rules) == 1 - assert isinstance(rules[0], RulePredicate) - assert format_ast == FormatAstRawText("hello world!") - - def test_when_is_parsed_in_the_context_of_a_question(self): - sheerka, context, parser = self.init_parser() - - text = "when foo is a bar print hello world" - res = parser.parse(context, ParserInput(text)) - format_rule = res.body.body - rules = format_rule.rule - - assert res.status - assert len(rules) == 1 - assert isinstance(rules[0], RulePredicate) - assert rules[0].predicate.body.body.get_metadata().pre == "is_question()" - - def test_when_can_support_multiple_possibilities(self): - sheerka, context, parser = self.init_parser() - - text = "when foo is good print hello world" - res = parser.parse(context, ParserInput(text)) - format_rule = res.body.body - rules = format_rule.rule - - assert res.status - assert len(rules) == 2 - assert rules[0].predicate.body.body.get_metadata().name == "a is good" - assert rules[1].predicate.body.body.get_metadata().name == "b is good" - - @pytest.mark.parametrize("text, error", [ - ("hello world", [KeywordNotFound(None, keywords=['when', 'print'])]), - ("when True", [KeywordNotFound([], keywords=['print'])]), - ("print True", [KeywordNotFound([], keywords=['when'])]), - ]) - def test_cannot_parse_when_not_for_me(self, text, error): - sheerka, context, parser = self.init_parser() - - res = parser.parse(context, ParserInput(text)) - not_for_me = res.body - - assert not res.status - assert sheerka.isinstance(not_for_me, BuiltinConcepts.NOT_FOR_ME) - assert not_for_me.reason == error - - @pytest.mark.parametrize("text, expected_error", [ - ("when a b print hello world!", BuiltinConcepts.TOO_MANY_ERRORS), - ]) - def test_i_cannot_parse_invalid_predicates(self, text, expected_error): - sheerka, context, parser = self.init_parser() - - res = parser.parse(context, ParserInput(text)) - - assert not res.status - assert sheerka.isinstance(res.body, expected_error) - - @pytest.mark.parametrize("expr, expected", [ - ("hello world", FormatAstRawText("hello world")), - ("{id}", FormatAstVariable("id")), - ]) - def test_i_can_parse_valid_print_expression(self, expr, expected): - sheerka, context, parser = self.init_parser() - - text = "when True print " + expr - res = parser.parse(context, ParserInput(text)) - format_rule = res.body.body - format_ast = format_rule.format_ast - - assert res.status - assert format_ast == expected diff --git a/tests/parsers/test_DefRuleParser.py b/tests/parsers/test_DefRuleParser.py new file mode 100644 index 0000000..3718237 --- /dev/null +++ b/tests/parsers/test_DefRuleParser.py @@ -0,0 +1,213 @@ +import pytest + +from core.builtin_concepts_ids import BuiltinConcepts +from core.concept import Concept +from core.sheerka.services.SheerkaExecute import ParserInput +from core.sheerka.services.SheerkaRuleManager import RuleCompiledPredicate, FormatAstNode +from core.tokenizer import Tokenizer, Keywords +from core.utils import tokens_are_matching +from parsers.BaseCustomGrammarParser import KeywordNotFound, NameNode, SyntaxErrorNode +from parsers.BaseParser import UnexpectedEofParsingError +from parsers.DefRuleParser import DefRuleParser, DefExecRuleNode, DefFormatRuleNode +from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka + +cmap = { + "foo": Concept("foo"), + "bar": Concept("bar"), + "is a": Concept("x is a y").def_var("x").def_var("y"), + "is a question": Concept("x is a y", pre="is_question()").def_var("x").def_var("y"), + "a is good": Concept("a is good", pre="is_question()").def_var("a"), + "b is good": Concept("b is good", pre="is_question()").def_var("b"), + "greetings": Concept("hello a").def_var("a") +} + + +class TestDefRuleParser(TestUsingMemoryBasedSheerka): + shared_ontology = None + + @classmethod + def setup_class(cls): + init_test_ref = cls().init_test(cache_only=False, ontology="#TestDefRuleParser#") + sheerka, context, *updated = init_test_ref.with_concepts(*cmap.values(), create_new=True).unpack() + for i, concept_name in enumerate(cmap): + cmap[concept_name] = updated[i] + + cls.shared_ontology = sheerka.get_ontology(context) + sheerka.pop_ontology(context) + + def init_parser(self, my_concepts_map=None, **kwargs): + if my_concepts_map is None: + sheerka, context = self.init_test().unpack() + sheerka.add_ontology(context, self.shared_ontology) + else: + sheerka, context, *updated = self.init_test().with_concepts(*my_concepts_map.values(), **kwargs).unpack() + for i, pair in enumerate(my_concepts_map): + my_concepts_map[pair] = updated[i] + + parser = DefRuleParser() + return sheerka, context, parser + + def test_i_cannot_parse_when_parser_input_is_initialized_from_token(self): + sheerka, context, parser = self.init_parser() + res = parser.parse(context, ParserInput("", list(Tokenizer("init from tokens")))) + + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.NOT_FOR_ME) + + def test_i_can_detect_empty_expression(self): + sheerka, context, parser = self.init_parser() + res = parser.parse(context, ParserInput("")) + + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.IS_EMPTY) + + def test_input_must_be_a_parser_input(self): + sheerka, context, parser = self.init_parser() + assert parser.parse(context, "not a parser input") is None + + def test_i_can_parse_simple_exec_rule_definition(self): + sheerka, context, parser = self.init_parser() + text = "when True then answer('that is true')" + + res = parser.parse(context, ParserInput(text)) + parser_result = res.body + parsed = res.body.body + + assert res.status + assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) + assert isinstance(parsed, DefExecRuleNode) + + assert len(parsed.tokens) == 2 + assert tokens_are_matching(parsed.tokens[Keywords.WHEN], Tokenizer("when True")) + assert tokens_are_matching(parsed.tokens[Keywords.THEN], Tokenizer("then answer('that is true')")) + assert isinstance(parsed.when, list) + assert len(parsed.when) == 1 + assert isinstance(parsed.when[0], RuleCompiledPredicate) + assert sheerka.isinstance(parsed.then, BuiltinConcepts.RETURN_VALUE) + + def test_i_can_parse_simple_format_rule_definition(self): + sheerka, context, parser = self.init_parser() + + text = "when True print hello world!" + res = parser.parse(context, ParserInput(text)) + parser_result = res.body + parsed = res.body.body + + assert res.status + assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) + assert isinstance(parsed, DefFormatRuleNode) + + assert len(parsed.tokens) == 2 + assert tokens_are_matching(parsed.tokens[Keywords.WHEN], Tokenizer("when True")) + assert tokens_are_matching(parsed.tokens[Keywords.PRINT], Tokenizer("print hello world!")) + assert isinstance(parsed.when, list) + assert len(parsed.when) == 1 + assert isinstance(parsed.when[0], RuleCompiledPredicate) + assert isinstance(parsed.print, FormatAstNode) + + def test_i_can_parse_exec_rule_with_name(self): + sheerka, context, parser = self.init_parser() + text = "def rule my rule as when True then answer('that is true')" + + res = parser.parse(context, ParserInput(text)) + parser_result = res.body + parsed = res.body.body + + assert res.status + assert sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) + assert isinstance(parsed, DefExecRuleNode) + + assert parsed.name == NameNode(list(Tokenizer("my rule"))) + assert len(parsed.tokens) == 2 + assert tokens_are_matching(parsed.tokens[Keywords.WHEN], Tokenizer("when True")) + assert tokens_are_matching(parsed.tokens[Keywords.THEN], Tokenizer("then answer('that is true')")) + assert isinstance(parsed.when, list) + assert len(parsed.when) == 1 + assert isinstance(parsed.when[0], RuleCompiledPredicate) + assert sheerka.isinstance(parsed.then, BuiltinConcepts.RETURN_VALUE) + + def test_when_is_parsed_in_the_context_of_a_question(self): + sheerka, context, parser = self.init_parser() + + text = "when foo is a bar print hello world" + res = parser.parse(context, ParserInput(text)) + format_rule = res.body.body + rules = format_rule.when + + assert res.status + assert len(rules) == 1 + assert isinstance(rules[0], RuleCompiledPredicate) + assert rules[0].predicate.body.body.get_metadata().pre == "is_question()" + + def test_when_can_support_multiple_possibilities_when_question_only(self): + sheerka, context, parser = self.init_parser() + + text = "when foo is good print hello world" + res = parser.parse(context, ParserInput(text)) + format_rule = res.body.body + rules = format_rule.when + + assert res.status + assert len(rules) == 2 + assert rules[0].predicate.body.body.get_metadata().name == "a is good" + assert rules[1].predicate.body.body.get_metadata().name == "b is good" + + @pytest.mark.parametrize("text, error", [ + ("def", [KeywordNotFound(None, keywords=['rule'])]), + ("def concept", [KeywordNotFound(None, keywords=['rule'])]), + ("def other", [KeywordNotFound(None, keywords=['rule'])]), + ("def rule name", [KeywordNotFound(None, keywords=['as'])]), + ("def rule complicated long name", [KeywordNotFound(None, keywords=['as'])]), + ("hello world", [KeywordNotFound(None, keywords=['when'])]), + ("when True", [KeywordNotFound([], keywords=['then', 'print'])]), + ("print True", [KeywordNotFound([], keywords=['when'])]), + ("when True print 'hello world' then answer('yes')", + [SyntaxErrorNode([], message="Cannot have both 'print' and 'then' keywords")]) + ]) + def test_i_can_detect_not_for_me(self, text, error): + sheerka, context, parser = self.init_parser() + + res = parser.parse(context, ParserInput(text)) + not_for_me = res.body + + assert not res.status + assert sheerka.isinstance(not_for_me, BuiltinConcepts.NOT_FOR_ME) + assert not_for_me.reason == error + + @pytest.mark.parametrize("text, expected_error", [ + ("when x x print 'hello world'", BuiltinConcepts.TOO_MANY_ERRORS), + + ]) + def test_i_can_detect_errors(self, text, expected_error): + sheerka, context, parser = self.init_parser() + + res = parser.parse(context, ParserInput(text)) + + assert not res.status + assert sheerka.isinstance(res.body, expected_error) + + @pytest.mark.parametrize("text, error_message", [ + ("def rule rule_name as", "While parsing 'when'."), + ("def rule rule_name as ", "While parsing 'when'."), + ]) + def test_i_cannot_parse_when_unexpected_eof(self, text, error_message): + sheerka, context, parser = self.init_parser() + + res = parser.parse(context, ParserInput(text)) + not_for_me = res.body + + assert not res.status + assert sheerka.isinstance(not_for_me, BuiltinConcepts.NOT_FOR_ME) + assert isinstance(not_for_me.reason[0], UnexpectedEofParsingError) + assert not_for_me.reason[0].message == error_message + + def test_rule_name_is_mandatory_when_using_def_rule(self): + sheerka, context, parser = self.init_parser() + + res = parser.parse(context, ParserInput("def rule as when True print 'true'")) + error = res.body + + assert not res.status + assert sheerka.isinstance(error, BuiltinConcepts.ERROR) + assert isinstance(error.error[0], SyntaxErrorNode) + assert error.error[0].message == "Name is mandatory" diff --git a/tests/parsers/test_ExpressionParser.py b/tests/parsers/test_ExpressionParser.py index 98d2bcc..c64e7b3 100644 --- a/tests/parsers/test_ExpressionParser.py +++ b/tests/parsers/test_ExpressionParser.py @@ -1,27 +1,21 @@ -from dataclasses import dataclass +import ast import pytest + from core.builtin_concepts import BuiltinConcepts, ReturnValueConcept -from core.concept import Concept +from core.concept import Concept, CMV, DoNotResolve, CC +from core.rule import Rule from core.sheerka.services.SheerkaExecute import ParserInput -from core.tokenizer import Tokenizer, TokenKind +from core.tokenizer import TokenKind +from parsers.BaseNodeParser import CNC from parsers.BaseParser import UnexpectedEofParsingError, UnexpectedTokenParsingError -from parsers.ExpressionParser import PropertyEqualsNode, PropertyEqualsSequenceNode, PropertyContainsNode, AndNode, \ - OrNode, NotNode, LambdaNode, IsaNode, NameExprNode, ExpressionParser, LeftPartNotFoundError, TrueifyVisitor - +from parsers.ExpressionParser import ExpressionParser, LeftPartNotFoundError, ParenthesisMismatchError +from parsers.PythonParser import PythonNode +from parsers.expressions import TrueifyVisitor, IsAQuestionVisitor, AndNode +from sheerkarete.network import ReteNetwork from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka - - -@dataclass -class Obj: - prop_a: object - prop_b: object = None - prop_c: object = None - parent: object = None - - -def n(value): - return NameExprNode(Tokenizer(value, yield_eof=False)) +from tests.parsers.parsers_utils import compute_expected_array, resolve_test_concept, EXPR, OR, AND, NOT, \ + get_expr_node_from_test_node, get_rete_conditions class TestExpressionParser(TestUsingMemoryBasedSheerka): @@ -32,19 +26,34 @@ class TestExpressionParser(TestUsingMemoryBasedSheerka): return sheerka, context, parser @pytest.mark.parametrize("expression, expected", [ - ("one complicated expression", n("one complicated expression")), - ("function_call(a,b,c)", n("function_call(a,b,c)")), - ("one expression or another expression", OrNode(n("one expression"), n("another expression"))), - ("one expression and another expression", AndNode(n("one expression"), n("another expression"))), - ("one or two or three", OrNode(n("one"), n("two"), n("three"))), - ("one and two and three", AndNode(n("one"), n("two"), n("three"))), - ("one or two and three", OrNode(n("one"), AndNode(n("two"), n("three")))), - ("one and two or three", OrNode(AndNode(n("one"), n("two")), n("three"))), - ("one and (two or three)", AndNode(n("one"), OrNode(n("two"), n("three")))), + ("one complicated expression", EXPR("one complicated expression")), + ("function_call(a,b,c)", EXPR("function_call(a,b,c)")), + ("one expression or another expression", OR(EXPR("one expression"), EXPR("another expression"))), + ("one expression and another expression", AND(EXPR("one expression"), EXPR("another expression"))), + ("not one", NOT(EXPR("one"))), + ("one and not two", AND(EXPR("one"), NOT(EXPR("two")))), + ("not one and two", AND(NOT(EXPR("one")), EXPR("two"))), + ("one or not two", OR(EXPR("one"), NOT(EXPR("two")))), + ("not one or two", OR(NOT(EXPR("one")), EXPR("two"))), + ("one or two or three", OR(EXPR("one"), EXPR("two"), EXPR("three"))), + ("one and two and three", AND(EXPR("one"), EXPR("two"), EXPR("three"))), + ("one or two and three", OR(EXPR("one"), AND(EXPR("two"), EXPR("three")))), + ("one and two or three", OR(AND(EXPR("one"), EXPR("two")), EXPR("three"))), + ("one and (two or three)", AND(EXPR("one"), OR(EXPR("two"), EXPR("three")), source="one and (two or three)")), + ("not not one", NOT(NOT(EXPR("one")))), + ("not (one and two)", NOT(AND(EXPR("one"), EXPR("two")), source="not (one and two)")), + ("one 'and' two or three", OR(EXPR("one 'and' two"), EXPR("three"))), + ("not ((a or b) and (c or d))", NOT(AND(OR(EXPR("a"), EXPR("b")), OR(EXPR("c"), EXPR("d")), + source="(a or b) and (c or d)"), + source="not ((a or b) and (c or d))")), + ("not ((a and b) or (c and d))", NOT(OR(AND(EXPR("a"), EXPR("b")), AND(EXPR("c"), EXPR("d")), + source="(a and b) or (c and d)"), + source="not ((a and b) or (c and d))")), ]) def test_i_can_parse_expression(self, expression, expected): sheerka, context, parser = self.init_parser() + expected = get_expr_node_from_test_node(expression, expected) res = parser.parse(context, ParserInput(expression)) wrapper = res.body expressions = res.body.body @@ -69,6 +78,15 @@ class TestExpressionParser(TestUsingMemoryBasedSheerka): assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) assert res.body.body == expected_errors + def test_i_can_detect_unexpected_not_error(self): + sheerka, context, parser = self.init_parser() + expression = "a cat is not a human" + + res = parser.parse(context, ParserInput(expression)) + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) + assert isinstance(res.body.body[0], UnexpectedTokenParsingError) + def test_i_can_detect_unbalanced_parenthesis(self): sheerka, context, parser = self.init_parser() @@ -86,6 +104,27 @@ class TestExpressionParser(TestUsingMemoryBasedSheerka): assert res.body.reason[0].token.type == TokenKind.RPAR assert res.body.reason[0].expected_tokens == [] + res = parser.parse(context, ParserInput("one and two(")) + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) + assert isinstance(res.body.body[0], ParenthesisMismatchError) + assert res.body.body[0].token.type == TokenKind.LPAR + assert res.body.body[0].token.index == 11 + + res = parser.parse(context, ParserInput("one (")) + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.NOT_FOR_ME) + assert isinstance(res.body.reason[0], ParenthesisMismatchError) + assert res.body.reason[0].token.type == TokenKind.LPAR + assert res.body.reason[0].token.index == 4 + + res = parser.parse(context, ParserInput("one (and")) + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) + assert isinstance(res.body.body[0], ParenthesisMismatchError) + assert res.body.body[0].token.type == TokenKind.LPAR + assert res.body.body[0].token.index == 4 + res = parser.parse(context, ParserInput("one and two)")) assert not res.status assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) @@ -93,7 +132,14 @@ class TestExpressionParser(TestUsingMemoryBasedSheerka): assert res.body.body[0].token.type == TokenKind.RPAR assert res.body.body[0].expected_tokens == [] - res = parser.parse(context, ParserInput("one and two)")) + res = parser.parse(context, ParserInput("one )")) + assert not res.status + assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) + assert isinstance(res.body.body[0], UnexpectedTokenParsingError) + assert res.body.body[0].token.type == TokenKind.RPAR + assert res.body.body[0].expected_tokens == [] + + res = parser.parse(context, ParserInput("one ) and")) assert not res.status assert sheerka.isinstance(res.body, BuiltinConcepts.ERROR) assert isinstance(res.body.body[0], UnexpectedTokenParsingError) @@ -107,90 +153,6 @@ class TestExpressionParser(TestUsingMemoryBasedSheerka): assert not res.status assert sheerka.isinstance(res.body, BuiltinConcepts.IS_EMPTY) - def test_i_can_test_property_equals(self): - node = PropertyEqualsNode("prop_a", "good value") - - assert node.eval(Obj(prop_a="good value")) - assert not node.eval(Obj(prop_a="other value")) - - def test_i_can_test_property_equals_for_int(self): - node = PropertyEqualsNode("prop_a", "1") - - assert node.eval(Obj(prop_a=1)) - assert node.eval(Obj(prop_a="1")) - - def test_i_can_test_property_equals_sequence(self): - node = PropertyEqualsSequenceNode(["prop_b", "prop_a"], ["good parent", "good child"]) - - assert node.eval(Obj(prop_a="good child", parent=Obj(prop_a="Don't care", prop_b="good parent"))) - assert not node.eval(Obj(prop_a="good child", parent=Obj(prop_a="Don't care", prop_b="wrong parent"))) - assert not node.eval(Obj(prop_a="good child")) - assert not node.eval(Obj(prop_a="wrong child", parent=Obj(prop_a="Don't care", prop_b="good parent"))) - - def test_i_can_test_property_contains(self): - node = PropertyContainsNode("prop_a", "substring") - - assert node.eval(Obj(prop_a="it contains substring in it")) - assert not node.eval(Obj(prop_a="it does not")) - - def test_i_can_test_property_contains_for_int(self): - node = PropertyContainsNode("prop_a", "44") - - assert node.eval(Obj(prop_a=123445)) - assert not node.eval(Obj(prop_a=12435)) - - def test_i_can_test_and(self): - left = PropertyEqualsNode("prop_a", "good a") - right = PropertyEqualsNode("prop_b", "good b") - other = PropertyEqualsNode("prop_c", "good c") - and_node = AndNode(left, right, other) - - assert and_node.eval(Obj("good a", "good b", "good c")) - assert not and_node.eval(Obj("wrong a", "good b", "good c")) - assert not and_node.eval(Obj("good a", "wrong b", "good c")) - assert not and_node.eval(Obj("good a", "good b", "wrong c")) - - def test_i_can_test_or(self): - left = PropertyEqualsNode("prop_a", "good a") - right = PropertyEqualsNode("prop_b", "good b") - other = PropertyEqualsNode("prop_c", "good c") - or_node = OrNode(left, right, other) - - assert or_node.eval(Obj("wrong a", "good b", "good c")) - assert or_node.eval(Obj("good a", "wrong b", "good c")) - assert or_node.eval(Obj("good a", "good b", "wrong c")) - assert not or_node.eval(Obj("wrong a", "wrong b", "wrong c")) - - def test_i_can_test_not(self): - node = PropertyEqualsNode("prop_a", "good value") - not_node = NotNode(node) - - assert not not_node.eval(Obj(prop_a="good value")) - assert not_node.eval(Obj(prop_a="wrong value")) - - def test_i_can_test_lambda_node(self): - node = LambdaNode(lambda o: o.prop_a + o.prop_b == "ab") - - assert node.eval(Obj(prop_a="a", prop_b="b")) - assert not node.eval(Obj(prop_a="wrong value", prop_b="wrong value")) - assert not node.eval(Obj(prop_a="wrong value")) # exception is caught - - def test_i_can_test_isa_node(self): - class_node = IsaNode(Obj) - assert class_node.eval(Obj(prop_a="value")) - assert not class_node.eval(TestExpressionParser()) - - concept_node = IsaNode(BuiltinConcepts.RETURN_VALUE) - assert concept_node.eval(ReturnValueConcept()) - assert concept_node.eval(Concept(name="foo", key=BuiltinConcepts.RETURN_VALUE)) - assert not concept_node.eval(Obj) - assert not concept_node.eval(Concept()) - - concept_node2 = IsaNode("foo") - assert concept_node2.eval(Concept("foo").init_key()) - assert not concept_node2.eval(Obj) - assert not concept_node2.eval(Concept()) - @pytest.mark.parametrize("expression, to_trueify, to_skip, expected", [ ("a", ["b"], ["a"], "a"), ("b", ["b"], ["a"], "True"), @@ -208,3 +170,321 @@ class TestExpressionParser(TestUsingMemoryBasedSheerka): translated_node = TrueifyVisitor(to_trueify, to_skip).visit(expr_node) assert str(translated_node) == expected + + @pytest.mark.parametrize("expression, expected", [ + ("foo", None), + ("", None), + ("is_question()", True), + (" is_question() ", True), + ("is_question ( ) ", True), + ("is _question()", None), + ("is_ question()", None), + ("is _ question()", None), + ("context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)", True), + ("not is_question()", False), + ("not context.in_context(BuiltinConcepts.EVAL_QUESTION_REQUESTED)", False), + ("not foo", None), + + ("not is_question() and not is_question()", False), + ("not is_question() and is_question()", False), + ("not is_question() and foo", False), + ("is_question() and not is_question()", False), + ("is_question() and is_question()", True), + ("is_question() and foo", True), + ("foo and not is_question()", False), + ("foo and is_question()", True), + ("foo and bar", None), + + ("not is_question() or not is_question()", False), + ("not is_question() or is_question()", True), + ("not is_question() or foo", False), + ("is_question() or not is_question()", True), + ("is_question() or is_question()", True), + ("is_question() or foo", True), + ("foo or not is_question()", False), + ("foo or is_question()", True), + ("foo or bar", None), + ]) + def test_is_a_question(self, expression, expected): + sheerka, context, parser = self.init_parser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + + assert IsAQuestionVisitor().visit(expr_node) == expected + + @pytest.mark.parametrize("expression, expected", [ + ("foo", "foo"), + ("one two", "one two"), + ("foo is a bar", CMV("is a", x='foo', y='bar')), + ("one two is a bar", [CNC("is a", source="one two is a bar", x="one two", y="bar")]), + ("foo is an foo bar", + [CNC("is an", source="foo is an foo bar", x=DoNotResolve(value='foo'), exclude_body=True)]), + ]) + def test_i_can_get_compiled_expr_from_simple_concepts_expressions(self, expression, expected): + concepts_map = { + "foo": Concept("foo"), + "bar": Concept("bar"), + "one two": Concept("one two"), + "is a": Concept("x is a y").def_var("x").def_var("y"), + "is an": Concept("x is an y", definition="('foo'|'bar')=x 'is an' 'foo bar'").def_var("x"), + } + sheerka, context, *concepts = self.init_test().with_concepts(*concepts_map.values(), create_new=True).unpack() + + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, [expr_node], "test") + + assert len(return_values) == 1 + ret = return_values[0] + + if isinstance(expected, list): + expected_nodes = compute_expected_array(concepts_map, expression, expected) + assert ret.body.body == expected_nodes + else: + expected_concept = resolve_test_concept(concepts_map, expected) + assert ret.body.body == expected_concept + + @pytest.mark.parametrize("expression", [ + "a == 5", + "foo > 5", + "func() == 5", + "not a == 5", + "not foo > 5", + "not func() == 5", + "isinstance(a, int)", + "func()", + "not isinstance(a, int)", + "not func()" + ]) + def test_i_can_get_compiled_expr_from_simple_python_expressions(self, expression): + sheerka, context, = self.init_test().unpack() + + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, [expr_node], "test") + + assert len(return_values) == 1 + ret = return_values[0] + + assert ret.status + python_node = ret.body.body.get_python_node() + _ast = ast.parse(expression, mode="eval") + expected_python_node = PythonNode(expression, _ast) + assert python_node == expected_python_node + + @pytest.mark.parametrize("expression", [ + "a and not b", + "not b and a", + "__ret and not __ret.status", + ]) + def test_i_can_compile_negative_conjunctions_when_pure_python(self, expression): + sheerka, context, *concepts = self.init_concepts("foo") + + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, expr_node.parts, "test") + + ast_ = ast.parse(expression, "", 'eval') + expected_python_node = PythonNode(expression, ast_) + + assert len(return_values) == 1 + ret = return_values[0] + + assert sheerka.objvalue(ret) == expected_python_node + + @pytest.mark.parametrize("expression, text_to_compile", [ + ("foo bar == 5", "__C__foo0bar__1001__C__ == 5"), + ("not foo bar == 5", "not __C__foo0bar__1001__C__ == 5"), + ]) + def test_i_can_get_compiled_expr_from_python_and_concept(self, expression, text_to_compile): + sheerka, context, *concepts = self.init_test().with_concepts(Concept("foo bar"), create_new=True).unpack() + + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, [expr_node], "test") + + assert len(return_values) == 1 + ret = return_values[0] + + assert ret.status + python_node = ret.body.body.get_python_node() + _ast = ast.parse(text_to_compile, mode="eval") + expected_python_node = PythonNode(text_to_compile, _ast, expression) + assert python_node == expected_python_node + + def test_i_can_get_compiled_expr_from__mix_of_concepts_and_python(self): + sheerka, context, animal, cat, dog, pet, is_a, is_an = self.init_test().with_concepts( + Concept("animal"), + Concept("a cat"), + Concept("dog"), + Concept("pet"), + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is an y", pre="is_question()", definition="('cat'|'bird')=x 'is an' animal").def_var("x"), + create_new=True + ).unpack() + + parser = ExpressionParser() + expression = "not a cat is a pet and not bird is an animal and not x > 5 and not dog is a pet" + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, expr_node.parts, "test") + + to_compile = 'not __C__00var0000is0a000var001__1005__C__' + to_compile += ' and not __C__00var0000is0an0y__1006__C__' + to_compile += ' and not x > 5' + to_compile += ' and not __C__00var0000is0a000var001__1005_1__C__' + ast_ = ast.parse(to_compile, "", 'eval') + expected_python_node = PythonNode(to_compile, ast_, expression) + + assert len(return_values) == 1 + ret = return_values[0] + python_node = ret.body.body + assert python_node == expected_python_node + assert python_node.objects == { + "__C__00var0000is0a000var001__1005__C__": CC(is_a, x=cat, y=pet), + "__C__00var0000is0an0y__1006__C__": CC(is_an, exclude_body=True, x=DoNotResolve("bird"), animal=animal), + "__C__00var0000is0a000var001__1005_1__C__": CMV(is_a, x="dog", y="pet"), + } + + def test_i_can_get_compiled_expr_from_mix(self): + sheerka, context, animal, cat, dog, pet, is_a, is_an = self.init_test().with_concepts( + Concept("animal"), + Concept("a cat"), + Concept("dog"), + Concept("pet"), + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is an y", pre="is_question()", definition="('cat'|'bird')=x 'is an' animal").def_var("x"), + create_new=True + ).unpack() + + expression = "a cat is a pet and bird is an animal and x > 5 and dog is a pet" + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, expr_node.parts, "test") + + assert len(return_values) == 1 + ret = return_values[0] + + to_compile = '__C__00var0000is0a000var001__1005__C__ and __C__00var0000is0an0y__1006__C__ and x > 5 and __C__00var0000is0a000var001__1005_1__C__' + ast_ = ast.parse(to_compile, "", 'eval') + expected_python_node = PythonNode(to_compile, ast_, expression) + + python_node = ret.body.body + assert python_node == expected_python_node + assert python_node.objects == { + "__C__00var0000is0a000var001__1005__C__": CC(is_a, x=cat, y=pet), + "__C__00var0000is0an0y__1006__C__": CC(is_an, exclude_body=True, x=DoNotResolve("bird"), animal=animal), + "__C__00var0000is0a000var001__1005_1__C__": CMV(is_a, x="dog", y="pet"), + } + + def test_i_can_get_compiled_expr_when_multiple_choices(self): + sheerka, context, *concepts = self.init_test().with_concepts( + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is a y", pre="is_question()", body="isa(x, y)").def_var("x").def_var("y"), + create_new=True + ).unpack() + + parser = ExpressionParser() + expression = "a is a b" + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, [expr_node], "test") + + assert len(return_values) == 2 + + ret = return_values[0] + assert sheerka.objvalue(ret)[0].concept == CMV(concepts[0], x="a", y="b") + + ret = return_values[1] + assert sheerka.objvalue(ret)[0].concept == CMV(concepts[1], x="a", y="b") + + def test_i_can_get_compiled_expr_from_mix_when_multiple_choices(self): + sheerka, context, *concepts = self.init_test().with_concepts( + Concept("animal"), + Concept("a cat"), + Concept("dog"), + Concept("pet"), + Concept("x is a y", pre="is_question()", body="isinstance(x, y)").def_var("x").def_var("y"), + Concept("x is a y", pre="is_question()", body="isa(x, y)").def_var("x").def_var("y"), + Concept("x is an y", pre="is_question()", definition="('cat'|'bird')=x 'is an' animal").def_var("x"), + create_new=True + ).unpack() + + expression = "a cat is a pet and bird is an animal and x > 5 and dog is a pet" + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + return_values, _ = parser.compile_conjunctions(context, expr_node.parts, "test") + + assert len(return_values) == 4 + trimmed_source = "a cat is a pet and bird is an animal and x > 5 and dog is a pet" + + current_ret = return_values[0] + python_source = "__C__00var0000is0a000var001__1005__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1005_1__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_ret) == resolved_expected + + current_ret = return_values[1] + assert sheerka.isinstance(current_ret, BuiltinConcepts.RETURN_VALUE) + python_source = "__C__00var0000is0a000var001__1005__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1006__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_ret) == resolved_expected + + current_ret = return_values[2] + assert sheerka.isinstance(current_ret, BuiltinConcepts.RETURN_VALUE) + python_source = "__C__00var0000is0a000var001__1006__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1005__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_ret) == resolved_expected + + current_ret = return_values[3] + python_source = "__C__00var0000is0a000var001__1006__C__ and __C__00var0000is0an0y__1007__C__ and x > 5 and __C__00var0000is0a000var001__1006_1__C__" + ast_ = ast.parse(python_source, "", 'eval') + resolved_expected = PythonNode(python_source, ast_, trimmed_source) + assert sheerka.objvalue(current_ret) == resolved_expected + + @pytest.mark.parametrize("expression, expected_conditions, test_obj", [ + ( + "__ret", + ["#__x_00__|__name__|'__ret'"], + ReturnValueConcept("Test", True, None) + ), + ( + "__ret.status == True", + ["#__x_00__|__name__|'__ret'", "#__x_00__|status|True"], + ReturnValueConcept("Test", True, None) + ), + ( + "__ret.status", + ["#__x_00__|__name__|'__ret'", "#__x_00__|status|True"], + ReturnValueConcept("Test", True, None) + ), + ( + "__ret and __ret.status", + ["#__x_00__|__name__|'__ret'", "#__x_00__|status|True"], + ReturnValueConcept("Test", True, None) + ), + ]) + def test_i_can_get_rete_condition_from_python(self, expression, expected_conditions, test_obj): + sheerka, context, = self.init_test().unpack() + expected_full_condition = get_rete_conditions(*expected_conditions) + + parser = ExpressionParser() + expr_node = parser.parse(context, ParserInput(expression)).body.body + + nodes = expr_node.parts if isinstance(expr_node, AndNode) else [expr_node] + _, rete_disjunctions = parser.compile_conjunctions(context, nodes, "test") + + assert len(rete_disjunctions) == 1 + assert rete_disjunctions == [expected_full_condition] + + # check against a Rete network + network = ReteNetwork() + rule = Rule("test", expression, None) + rule.metadata.id = 9999 + rule.metadata.is_compiled = True + rule.metadata.is_enabled = True + rule.rete_disjunctions = rete_disjunctions + network.add_rule(rule) + + network.add_obj("__ret", test_obj) + matches = list(network.matches) + assert len(matches) > 0 diff --git a/tests/parsers/test_FunctionParser.py b/tests/parsers/test_FunctionParser.py index c074026..33d2aa7 100644 --- a/tests/parsers/test_FunctionParser.py +++ b/tests/parsers/test_FunctionParser.py @@ -28,7 +28,7 @@ class TestFunctionParser(TestUsingMemoryBasedSheerka): cmap[concept_name] = updated[i] cls.shared_ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() + sheerka.pop_ontology(context) def init_parser(self, my_concepts_map=None, **kwargs): if my_concepts_map is None: diff --git a/tests/parsers/test_PythonWithConceptsParser.py b/tests/parsers/test_PythonWithConceptsParser.py index 855d2fe..c5af4b9 100644 --- a/tests/parsers/test_PythonWithConceptsParser.py +++ b/tests/parsers/test_PythonWithConceptsParser.py @@ -1,18 +1,18 @@ import ast import pytest -import core.utils + from core.builtin_concepts import ParserResultConcept, BuiltinConcepts, ReturnValueConcept from core.concept import Concept from core.rule import Rule from core.sheerka.services.SheerkaExecute import ParserInput from core.tokenizer import Token, TokenKind, Tokenizer -from parsers.BaseNodeParser import ConceptNode, UnrecognizedTokensNode, RuleNode +from parsers.BaseNodeParser import ConceptNode, UnrecognizedTokensNode, RuleNode, SourceCodeNode from parsers.PythonParser import PythonNode from parsers.PythonWithConceptsParser import PythonWithConceptsParser from parsers.UnrecognizedNodeParser import UnrecognizedNodeParser - from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka +from tests.parsers.parsers_utils import get_source_code_node unrecognized_nodes_parser = UnrecognizedNodeParser() @@ -73,7 +73,8 @@ class TestPythonWithConceptsParser(TestUsingMemoryBasedSheerka): assert context.sheerka.isinstance(wrapper, BuiltinConcepts.PARSER_RESULT) assert wrapper.source == "foo + 1" assert isinstance(return_value, PythonNode) - assert return_value.source == "foo + 1" + assert return_value.source == "__C__foo__C__ + 1" + assert return_value.original_source == "foo + 1" assert return_value.get_dump(return_value.ast_) == to_str_ast("__C__foo__C__ + 1") assert return_value.objects["__C__foo__C__"] == foo @@ -93,7 +94,8 @@ class TestPythonWithConceptsParser(TestUsingMemoryBasedSheerka): assert context.sheerka.isinstance(wrapper, BuiltinConcepts.PARSER_RESULT) assert wrapper.source == "foo + 1" assert isinstance(return_value, PythonNode) - assert return_value.source == "foo + 1" + assert return_value.source == "__C__foo__1001__C__ + 1" + assert return_value.original_source == "foo + 1" assert return_value.get_dump(return_value.ast_) == to_str_ast("__C__foo__1001__C__ + 1") assert return_value.objects["__C__foo__1001__C__"] == foo @@ -124,7 +126,8 @@ class TestPythonWithConceptsParser(TestUsingMemoryBasedSheerka): assert context.sheerka.isinstance(parser_result, BuiltinConcepts.PARSER_RESULT) assert parser_result.source == "func(foo, bar)" assert isinstance(return_value, PythonNode) - assert return_value.source == "func(foo, bar)" + assert return_value.source == "func(__C__foo__1001__C__, __C__bar__1002__C__)" + assert return_value.original_source == "func(foo, bar)" assert return_value.get_dump(return_value.ast_) == to_str_ast("func(__C__foo__1001__C__, __C__bar__1002__C__)") assert return_value.objects["__C__foo__1001__C__"] == foo assert return_value.objects["__C__bar__1002__C__"] == bar @@ -144,7 +147,8 @@ class TestPythonWithConceptsParser(TestUsingMemoryBasedSheerka): assert context.sheerka.isinstance(wrapper, BuiltinConcepts.PARSER_RESULT) assert wrapper.source == "r:|rule_id: + 1" assert isinstance(return_value, PythonNode) - assert return_value.source == "r:|rule_id: + 1" + assert return_value.source == "__R____rule_id__R__ + 1" + assert return_value.original_source == "r:|rule_id: + 1" assert return_value.get_dump(return_value.ast_) == to_str_ast("__R____rule_id__R__ + 1") assert return_value.objects["__R____rule_id__R__"] == rule @@ -193,3 +197,26 @@ class TestPythonWithConceptsParser(TestUsingMemoryBasedSheerka): assert not result.status assert context.sheerka.isinstance(result.value, BuiltinConcepts.NOT_FOR_ME) + + def test_i_can_parse_nodes_when_source_code_node(self): + context = self.get_context() + foo = Concept("foo") + bar = Concept("bar") + + nodes = [ + UnrecognizedTokensNode(0, 1, list(Tokenizer("not ", yield_eof=False))), + get_source_code_node(2, "foo == 1", {"foo": foo}), + UnrecognizedTokensNode(7, 9, list(Tokenizer(" and ", yield_eof=False))), + get_source_code_node(10, "bar < 1", {"bar": bar}), + ] + expected_ast = ast.parse('not __C__foo__C__ == 1 and __C__bar__C__ < 1', "", 'eval') + + parser = PythonWithConceptsParser() + result = parser.parse_nodes(context, nodes) + + result_python_node = result.value.value + assert isinstance(result_python_node, PythonNode) + assert result_python_node.source == 'not __C__foo__C__ == 1 and __C__bar__C__ < 1' + assert result_python_node.ast_str == PythonNode.get_dump(expected_ast) + assert result_python_node.original_source == "not foo == 1 and bar < 1" + assert result_python_node.objects == {"__C__foo__C__": foo, "__C__bar__C__": bar} diff --git a/tests/parsers/test_RuleParser.py b/tests/parsers/test_RuleParser.py index 1d729a0..4819e69 100644 --- a/tests/parsers/test_RuleParser.py +++ b/tests/parsers/test_RuleParser.py @@ -14,17 +14,17 @@ class TestRuleParser(TestUsingMemoryBasedSheerka): @classmethod def setup_class(cls): init_test_helper = cls().init_test(cache_only=False, ontology="#TestRuleParser#") - sheerka, context, *updated = init_test_helper.with_rules(*my_rules).unpack() + sheerka, context, *updated = init_test_helper.with_format_rules(*my_rules).unpack() cls.shared_ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() + sheerka.pop_ontology(context) def init_parser(self, rules=None, **kwargs): if rules is None: sheerka, context = self.init_test().unpack() sheerka.add_ontology(context, self.shared_ontology) else: - sheerka, context, *updated = self.init_test().with_rules(*rules, **kwargs).unpack() + sheerka, context, *updated = self.init_test().with_format_rules(*rules, **kwargs).unpack() parser = RuleParser() return sheerka, context, parser diff --git a/tests/parsers/test_SyaNodeParser.py b/tests/parsers/test_SyaNodeParser.py index 0040ce5..0ddd0d9 100644 --- a/tests/parsers/test_SyaNodeParser.py +++ b/tests/parsers/test_SyaNodeParser.py @@ -66,7 +66,7 @@ class TestSyaNodeParser(TestUsingMemoryBasedSheerka): CONCEPT_COMPARISON_CONTEXT) cls.shared_ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() + sheerka.pop_ontology(context) def init_parser(self, my_concepts_map=None, diff --git a/tests/parsers/test_UnrecognizedNodeParser.py b/tests/parsers/test_UnrecognizedNodeParser.py index 79b494e..2a6a207 100644 --- a/tests/parsers/test_UnrecognizedNodeParser.py +++ b/tests/parsers/test_UnrecognizedNodeParser.py @@ -85,7 +85,7 @@ class TestUnrecognizedNodeParser(TestUsingMemoryBasedSheerka): concepts_map["plus"], 'Sya') cls.shared_ontology = sheerka.get_ontology(context) - sheerka.pop_ontology() + sheerka.pop_ontology(context) def init_parser(self, my_concepts_map=None, **kwargs): if my_concepts_map is None: diff --git a/tests/sheerkapickle/test_SheerkaPickler.py b/tests/sheerkapickle/test_SheerkaPickler.py index 9958ffb..775e071 100644 --- a/tests/sheerkapickle/test_SheerkaPickler.py +++ b/tests/sheerkapickle/test_SheerkaPickler.py @@ -31,6 +31,14 @@ class Obj: class TestSheerkaPickler(TestUsingMemoryBasedSheerka): + user_input_id = 0 + return_value_id = 0 + + @classmethod + def setup(cls): + sheerka = cls().get_sheerka() + cls.user_input_id = sheerka.get_by_key("__USER_INPUT").id + cls.return_value_id = sheerka.get_by_key("__RETURN_VALUE").id @pytest.mark.parametrize("obj, expected", [ (1, 1), @@ -194,13 +202,13 @@ class TestSheerkaPickler(TestUsingMemoryBasedSheerka): sheerka, context = self.init_test().unpack() sheerka.push_ontology(context, "new ontology") - ontology = sheerka.pop_ontology().body.body + ontology = sheerka.pop_ontology(context).body.body obj = sheerka.ret(sheerka.name, True, ontology) flatten = SheerkaPickler(sheerka).flatten(obj) assert flatten == { '_sheerka/obj': 'core.builtin_concepts.ReturnValueConcept', - 'concept/id': ('__RETURN_VALUE', '43'), + 'concept/id': ('__RETURN_VALUE', self.return_value_id), 'status': True, 'value': 'new ontology', 'who': '__SHEERKA'} diff --git a/tests/sheerkarete/RuleForTestingRete.py b/tests/sheerkarete/RuleForTestingRete.py new file mode 100644 index 0000000..88dedaa --- /dev/null +++ b/tests/sheerkarete/RuleForTestingRete.py @@ -0,0 +1,24 @@ +from core.rule import Rule, ACTION_TYPE_TEST + + +class RuleForTestingRete(Rule): + def __init__(self, *disjunctions, name=None): + """ + :param disjunctions: list of list of Condition. List of OR. inner list is a list of AND + :para name: optional name for the rule + """ + try: + predicate = " or ".join([ + " and ".join([str(c) for c in disjunction.conditions]) for disjunction in + disjunctions]) + except AttributeError: + predicate = "N/A" + + super().__init__(ACTION_TYPE_TEST, name, predicate) + self.disjunctions = disjunctions + self.metadata.id = 9999 + self.metadata.is_compiled = True + self.metadata.is_enabled = True + + def get_rete_disjunctions(self): + return self.disjunctions diff --git a/tests/sheerkarete/__init__.py b/tests/sheerkarete/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/sheerkarete/test_common.py b/tests/sheerkarete/test_common.py new file mode 100644 index 0000000..f534246 --- /dev/null +++ b/tests/sheerkarete/test_common.py @@ -0,0 +1,31 @@ +from sheerkarete.common import ReteToken, WME, V +from sheerkarete.conditions import Condition, NegatedConjunctiveConditions + + +def test_token(): + tdummy = ReteToken(None, None) + t0 = ReteToken(tdummy, WME('B1', 'on', 'B2')) + + assert tdummy.parent is None + assert t0.parent == tdummy + + assert tdummy.children == [t0] + assert t0.children == [] + + +def test_condition_vars(): + c0 = Condition(V('x'), 'is', V('y')) + assert c0.vars == [('identifier', V("x")), ('value', V("y"))] + + +def test_condition_contain(): + c0 = Condition(V('a'), V('b'), V('c')) + assert c0.contain(V('a')) + assert not c0.contain(V('d')) + + +def test_ncc(): + c0 = Condition(V('a'), V('b'), V('c')) + c1 = NegatedConjunctiveConditions(Condition(V('x'), 'color', 'red')) + c2 = NegatedConjunctiveConditions(c0, c1) + assert c2.number_of_conditions == 2 diff --git a/tests/sheerkarete/test_conditions.py b/tests/sheerkarete/test_conditions.py new file mode 100644 index 0000000..8199a1b --- /dev/null +++ b/tests/sheerkarete/test_conditions.py @@ -0,0 +1,341 @@ +from sheerkarete.common import WME, V +from sheerkarete.conditions import NotEqualsCondition, AndConditions, Condition, NegatedCondition, \ + NegatedConjunctiveConditions, FilterCondition, BindCondition +from sheerkarete.network import ReteNetwork +from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka +from tests.sheerkarete.RuleForTestingRete import RuleForTestingRete + + +class TestReteConditions(TestUsingMemoryBasedSheerka): + def test_i_can_manage_condition(self): + network = ReteNetwork() + condition = Condition("id", "attr", "value") + rule = RuleForTestingRete(AndConditions([condition])) + network.add_rule(rule) + + assert len(list(network.matches)) == 0 + + wme = WME("id", "attr", "value") + network.add_wme(wme) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + assert matches[0].token.wmes == [wme] + + # remove wme + network.remove_wme(wme) + assert len(list(network.matches)) == 0 + + def test_i_can_manage_not_equals_condition(self): + network = ReteNetwork() + + # I can add the condition + cond1 = NotEqualsCondition("id", "value", 5) + rule1 = RuleForTestingRete(AndConditions([cond1]), name="first id != 5") + network.add_rule(rule1) + + assert list(network.alpha_hash.keys()) == [("id", "value", "*")] + assert len(network.alpha_hash[("id", "value", "*")]) == 1 + assert network.alpha_hash[("id", "value", "*")][0].condition == cond1 + + # Another rule with the same condition + rule2 = RuleForTestingRete(AndConditions([cond1]), name="second id != 5") + network.add_rule(rule2) + assert list(network.alpha_hash.keys()) == [("id", "value", "*")] + assert len(network.alpha_hash[("id", "value", "*")]) == 1 + assert network.alpha_hash[("id", "value", "*")][0].condition == cond1 + + cond2 = NotEqualsCondition("id", "value", 6) + rule3 = RuleForTestingRete(AndConditions([cond2]), name="id != 6") + network.add_rule(rule3) + assert list(network.alpha_hash.keys()) == [("id", "value", "*")] + assert len(network.alpha_hash[("id", "value", "*")]) == 2 + assert network.alpha_hash[("id", "value", "*")][0].condition == cond1 + assert network.alpha_hash[("id", "value", "*")][1].condition == cond2 + + # I can match + wme1 = WME("id", "value", "1") + network.add_wme(wme1) + + matches = list(network.matches) + assert len(matches) == 2 + assert matches[0].pnode.rules == [rule1, rule2] + assert matches[1].pnode.rules == [rule3] + + # I can remove + network.remove_wme(wme1) + assert len(list(network.matches)) == 0 + + # I can add a non match + wme2 = WME("id", "value", 5) + network.add_wme(wme2) + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule3] + + def test_i_can_manage_negative_condition(self): + network = ReteNetwork() + condition = NegatedCondition("id", "attr", "value") + rule = RuleForTestingRete(AndConditions([condition])) + network.add_rule(rule) + + assert len(network.beta_root.children) == 1 + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + assert matches[0].token.wmes == [None] + + wme = WME("id", "attr", "value") + network.add_wme(wme) + matches = list(network.matches) + assert len(matches) == 0 + + # remove wme + network.remove_wme(wme) + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + assert matches[0].token.wmes == [None] + + def test_i_can_manage_ncc(self): + network = ReteNetwork() + c1 = Condition(V("x"), "is a", "light") + c2 = Condition(V("x"), "is", "red") + c3 = Condition(V("x"), "is", "blue") + rule = RuleForTestingRete(AndConditions([c1, NegatedConjunctiveConditions(c2, c3)])) + network.add_rule(rule) + + wme1 = WME("x1", "is a", "light") + wme2 = WME("x1", "is", "blue") + network.add_wme(wme1) + network.add_wme(wme2) + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme1, None] + + wme3 = WME("x1", "is", "red") + network.add_wme(wme3) + assert len(list(network.matches)) == 0 + + wme4 = WME("x2", "is a", "light") + network.add_wme(wme4) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme4, None] + + # remove a wme + network.remove_wme(wme2) + matches = list(network.matches) + assert len(matches) == 2 + assert matches[0].token.wmes == [wme4, None] + assert matches[1].token.wmes == [wme1, None] + + def test_i_can_manage_ncc_when_already_inserted_elements(self): + network = ReteNetwork() + + wme1 = WME("x1", "is a", "light") + wme2 = WME("x1", "is", "blue") + network.add_wme(wme1) + network.add_wme(wme2) + + c1 = Condition(V("x"), "is a", "light") + c2 = Condition(V("x"), "is", "red") + c3 = Condition(V("x"), "is", "blue") + rule = RuleForTestingRete(AndConditions([c1, NegatedConjunctiveConditions(c2, c3)])) + network.add_rule(rule) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme1, None] + + wme3 = WME("x1", "is", "red") + network.add_wme(wme3) + assert len(list(network.matches)) == 0 + + # remove back + network.remove_wme(wme2) + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme1, None] + + def test_i_can_manage_ncc_when_already_inserted_elements2(self): + network = ReteNetwork() + + wme1 = WME("x1", "is a", "light") + wme2 = WME("x1", "is", "blue") + wme3 = WME("x1", "is", "red") + network.add_wme(wme1) + network.add_wme(wme2) + network.add_wme(wme3) + + c1 = Condition(V("x"), "is a", "light") + c2 = Condition(V("x"), "is", "red") + c3 = Condition(V("x"), "is", "blue") + rule = RuleForTestingRete(AndConditions([c1, NegatedConjunctiveConditions(c2, c3)])) + network.add_rule(rule) + assert len(list(network.matches)) == 0 + + # remove a wme + network.remove_wme(wme3) + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme1, None] + + def test_i_can_manage_filter_condition(self): + network = ReteNetwork() + c1 = Condition("value", "equals", V("x")) + c2 = Condition("value", "equals", V("y")) + c3 = FilterCondition(lambda x, y: x == y + 1) + rule = RuleForTestingRete(AndConditions([c1, c2, c3])) + network.add_rule(rule) + + assert len(list(network.matches)) == 0 + + wme1 = WME("value", "equals", 5) + wme2 = WME("value", "equals", 3) + network.add_wme(wme1) + network.add_wme(wme2) + assert len(list(network.matches)) == 0 + + wme3 = WME("value", "equals", 4) + network.add_wme(wme3) + matches = list(network.matches) + assert len(matches) == 2 + assert matches[0].token.wmes == [wme1, wme3] + assert matches[1].token.wmes == [wme3, wme2] + + # remove a WME + network.remove_wme(wme1) + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme3, wme2] + + def test_i_can_manage_filter_when_it_is_the_first_condition(self): + network = ReteNetwork() + c1 = FilterCondition(lambda: True) + rule = RuleForTestingRete(AndConditions([c1])) + network.add_rule(rule) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [None] + assert matches[0].token.binding == {} + + # remove wme + network.remove_rule(rule) + assert len(list(network.matches)) == 0 + + def test_i_can_filter_already_inserted_elements(self): + network = ReteNetwork() + wme1 = WME("value", "equals", 5) + wme2 = WME("value", "equals", 3) + wme3 = WME("value", "equals", 4) + network.add_wme(wme1) + network.add_wme(wme2) + network.add_wme(wme3) + + c1 = Condition("value", "equals", V("x")) + c2 = Condition("value", "equals", V("y")) + c3 = FilterCondition(lambda x, y: x == y + 1) + rule = RuleForTestingRete(AndConditions([c1, c2, c3])) + network.add_rule(rule) + + matches = list(network.matches) + assert len(matches) == 2 + assert matches[0].token.wmes == [wme3, wme2] + assert matches[1].token.wmes == [wme1, wme3] + + def test_i_can_manage_binding_conditions(self): + network = ReteNetwork() + c1 = Condition("value", "equals", V("x")) + c2 = BindCondition(lambda x: x + 5, V("z")) + rule = RuleForTestingRete(AndConditions([c1, c2])) + network.add_rule(rule) + + assert len(list(network.matches)) == 0 + + wme1 = WME("value", "equals", 1) + network.add_wme(wme1) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [wme1] + assert matches[0].token.binding == {V("z"): 6, V("x"): 1} + + # remove wme + network.remove_wme(wme1) + assert len(list(network.matches)) == 0 + + def test_i_can_manage_bind_when_it_is_the_first_condition(self): + network = ReteNetwork() + c1 = BindCondition(lambda: 5, V("x")) + rule = RuleForTestingRete(AndConditions([c1])) + network.add_rule(rule) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].token.wmes == [None] + assert matches[0].token.binding == {V("x"): 5} + + # remove wme + network.remove_rule(rule) + assert len(list(network.matches)) == 0 + + def test_i_can_match_on_the_correct_variable_name(self): + network = ReteNetwork() + condition = Condition(V("x"), "attr", V("x")) + rule = RuleForTestingRete(AndConditions([condition])) + network.add_rule(rule) + + wme = WME('a', 'attr', 'c') + network.add_wme(wme) + assert len(list(network.matches)) == 0 + + wme = WME('a', 'attr', 'a') + network.add_wme(wme) + assert len(list(network.matches)) == 1 + + def test_condition_test(self): + c0 = Condition(V('x'), 'color', 'red') + w0 = WME('B1', 'color', 'red') + w1 = WME('B1', 'color', 'blue') + assert c0.test(w0) + assert not c0.test(w1) + + c1 = Condition(V('x'), 'is', V('x')) + w2 = WME('B1', 'is', 'B3') + assert not c1.test(w2) + + def test_black_white(self): + # KSI 2021-02-07. I simply copy this test + + net = ReteNetwork() + c1 = Condition(V('item'), 'cat', V('cid')) + c2 = Condition(V('item'), 'shop', V('sid')) + white = NegatedConjunctiveConditions( + NegatedCondition(V('item'), 'cat', '100'), + NegatedCondition(V('item'), 'cat', '101'), + NegatedCondition(V('item'), 'cat', '102'), + ) + n1 = NegatedCondition(V('item'), 'shop', '1') + n2 = NegatedCondition(V('item'), 'shop', '2') + n3 = NegatedCondition(V('item'), 'shop', '3') + + rule = RuleForTestingRete(AndConditions([c1, c2, white, n1, n2, n3])) + net.add_rule(rule) + + wmes = [ + WME('item:1', 'cat', '101'), + WME('item:1', 'shop', '4'), + WME('item:2', 'cat', '100'), + WME('item:2', 'shop', '1'), + ] + for wme in wmes: + net.add_wme(wme) + + p0 = rule.rete_p_nodes[0] + assert len(list(p0.activations)) == 1 + assert list(p0.activations)[0].binding[V('item')] == 'item:1' diff --git a/tests/sheerkarete/test_network.py b/tests/sheerkarete/test_network.py new file mode 100644 index 0000000..875730b --- /dev/null +++ b/tests/sheerkarete/test_network.py @@ -0,0 +1,685 @@ +import pytest + +from core.builtin_concepts import ReturnValueConcept +from core.concept import Concept, DEFINITION_TYPE_DEF +from core.rule import Rule, ACTION_TYPE_EXEC +from core.sheerka.services.SheerkaEvaluateRules import SheerkaEvaluateRules +from core.sheerka.services.SheerkaRuleManager import SheerkaRuleManager +from sheerkarete.beta import BetaMemory +from sheerkarete.common import V, WME, ReteToken +from sheerkarete.conditions import Condition, NegatedCondition, AndConditions +from sheerkarete.join_node import JoinNode +from sheerkarete.negative_node import NegativeNode +from sheerkarete.network import ReteNetwork, FACT_ID +from tests.TestUsingMemoryBasedSheerka import TestUsingMemoryBasedSheerka +from tests.sheerkarete.RuleForTestingRete import RuleForTestingRete + + +class TestReteNetwork(TestUsingMemoryBasedSheerka): + @pytest.mark.parametrize("condition, expected_key", [ + (Condition("id", "attr", "value"), ("id", "attr", "value")), + (Condition(V("x"), "attr", "value"), ("*", "attr", "value")), + (Condition("id", V("x"), "value"), ("id", "*", "value")), + (Condition("id", "attr", V("x")), ("id", "attr", "*")), + (Condition(V("x"), V("x"), V("x")), ("*", "*", "*")), + ]) + def test_i_can_create_alpha_memory(self, condition, expected_key): + network = ReteNetwork() + + amem = network.build_or_share_alpha_memory(condition) + assert amem.key == expected_key + assert amem.key in network.alpha_hash + + def test_same_amem_is_reused(self): + network = ReteNetwork() + condition = Condition("id", "attr", "value") + + amem1 = network.build_or_share_alpha_memory(condition) + amem2 = network.build_or_share_alpha_memory(condition) + + assert amem1 == amem2 + assert len(network.alpha_hash) == 1 + + def test_i_can_initialize_a_simple_network(self): + network = ReteNetwork() + condition = Condition("id", "attr", "value") + rule = RuleForTestingRete(AndConditions([condition])) + network.add_rule(rule) + + assert condition.get_key() in network.alpha_hash + assert len(network.pnodes) == 1 + + join_node = network.pnodes[0].parent + assert isinstance(join_node, JoinNode) + assert isinstance(join_node.parent, BetaMemory) + assert join_node.amem == network.alpha_hash[condition.get_key()][0] + + def test_i_can_add_multiple_rules_with_the_same_conditions(self): + network = ReteNetwork() + condition = Condition("id", "attr", "value") + rule1 = RuleForTestingRete(AndConditions([condition])) + rule2 = RuleForTestingRete(AndConditions([condition])) + network.add_rule(rule1) + network.add_rule(rule2) + + assert len(network.pnodes) == 1 + assert network.pnodes[0].rules == [rule1, rule2] + + def test_i_can_update_conditions_attributes_by_id_when_constraint_on__name__(self): + network = ReteNetwork() + conditions = [Condition(V("x"), "__name__", "fact_name"), + Condition(V("x"), "attr1", "value1"), + Condition(V("x"), "attr2", "value1")] + + rule = RuleForTestingRete(AndConditions(conditions)) + network.add_rule(rule) + + assert network.attributes_by_id == {"fact_name": ["__name__", "attr1", "attr2"]} + + def test_adding_obj_when_no_rule_has_no_effect(self): + network = ReteNetwork() + ret = ReturnValueConcept("test", True, "value") + network.add_obj("__ret", ret) + + assert network.working_memory == set() + + def test_adding_obj_when_rule_with_no_attribute_constraint(self): + """ + When a rule has no attribute constraint, we are forced to provide all the known properties + """ + network = ReteNetwork() + ret = ReturnValueConcept("test", True, "value") + network.add_rule(RuleForTestingRete( + AndConditions([Condition("__ret", "*", True)]))) # test only, this rule cannot be matched + + network.add_obj("__ret", ret) + assert network.working_memory == { + WME("f-00000", "status", True), + WME("f-00000", "value", "value"), + WME("f-00000", "body", "value"), + WME("f-00000", "parents", None), + WME("f-00000", "id", None), + WME("f-00000", "name", "__RETURN_VALUE"), + WME("f-00000", "key", "__RETURN_VALUE"), + WME("f-00000", "self", ret), + WME("f-00000", "who", "test"), + } + + def test_adding_obj_when_rule_with_attributes_constraints(self): + """ + When a rule with attribute constraint, we only add the requested attributes + """ + network = ReteNetwork() + ret = ReturnValueConcept("test", True, "value") + network.add_rule(RuleForTestingRete( + AndConditions([Condition("__ret", "status", True), Condition("__ret", "body", "value")]))) + + network.add_obj("__ret", ret) + assert network.working_memory == { + WME("f-00000", "status", True), + WME("f-00000", "body", "value"), + } + + def test_adding_obj_when_requested_attribute_is_not_found(self): + """ + When a rule with attribute constraint, we only add the requested attributes + """ + network = ReteNetwork() + ret = ReturnValueConcept("test", True, "value") + network.add_rule(RuleForTestingRete(AndConditions([Condition("__ret", "not found", True), + Condition("__ret", "body", "value")]))) + + network.add_obj("__ret", ret) + assert network.working_memory == { + WME("f-00000", "body", "value"), + } + + def test_adding_object_when_id_is_not_defined_but_attribute_is_known(self): + network = ReteNetwork() + rule = RuleForTestingRete(AndConditions([ + Condition(V("1"), "status", True), + ])) + network.add_rule(rule) + + ret1 = ReturnValueConcept("test", True, "first one") + ret2 = ReturnValueConcept("test", False, "to discard") + + network.add_obj("__ret", ret1) + network.add_obj("__ret", ret2) + assert network.working_memory == { + WME("f-00000", "status", True), + WME("f-00001", "status", False), + } + + def test_adding_obj_full_test_obj_is_a_concept(self): + """ + When the value is a concept, we create sub objects, to ease the requests + """ + sheerka, context, boy, the, greetings = self.init_concepts( + "boy", + Concept("the x").def_var("x"), + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a") + ) + network = ReteNetwork() + + the_boy = sheerka.new(the, x=boy) + hello_the_boy = sheerka.new(greetings, a=the_boy) + + # original condition may look like + # __ret.status + # and __ret.body == $body + # and isinstance($body, Concept) + # and $body.name == "greetings" + # and $body.get_value("a") == $the + # and isinstance($the, Concept) + # and $the.id == 1002 + # and $the.get_value("x") = $boy + # and instance($boy, Concept) + # and $boy.name = "boy" + ret = ReturnValueConcept("test", True, hello_the_boy) + rule = RuleForTestingRete(AndConditions([Condition(V("ret"), "__name__", "__ret"), + Condition(V("ret"), "status", True), + Condition(V("ret"), "body", V("body")), + Condition(V("body"), "name", "greetings"), + Condition(V("body"), "a", V("the")), + Condition(V("the"), "id", '1002'), + Condition(V("the"), "x", V("boy")), + Condition(V("boy"), "name", "boy"), + ])) + network.add_rule(rule) + + network.add_obj("__ret", ret) + assert network.working_memory == { + WME("f-00000", "__name__", "__ret"), + WME("f-00000", "status", True), + WME("f-00000", "body", "f-00000.body"), + WME("f-00000.body", "id", "1003"), + WME("f-00000.body", "name", "greetings"), + WME("f-00000.body", "key", "hello __var__0"), + WME("f-00000.body", "a", "f-00000.body.a"), + WME("f-00000.body", "self", ret.body), + WME("f-00000.body.a", "id", "1002"), + WME("f-00000.body.a", "name", "the x"), + WME("f-00000.body.a", "key", "the __var__0"), + WME("f-00000.body.a", "x", "f-00000.body.a.x"), + WME("f-00000.body.a", "self", the_boy), + WME("f-00000.body.a.x", "id", "1001"), + WME("f-00000.body.a.x", "name", "boy"), + WME("f-00000.body.a.x", "key", "boy"), + WME("f-00000.body.a.x", "self", boy), + } + + # sanity check that the WME produced match the condition + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + assert network.facts == {'f-00000': ret} + + def test_i_can_distinguish_objects_with_different_value(self): + network = ReteNetwork() + rule = RuleForTestingRete(AndConditions([ + Condition(V("1"), "status", True), + Condition(V("1"), "value", "first one"), + Condition(V("2"), "status", True), + Condition(V("2"), "value", "second one") + ])) + network.add_rule(rule) + + ret1 = ReturnValueConcept("test", True, "first one") + ret2 = ReturnValueConcept("test", False, "to discard") + ret3 = ReturnValueConcept("test", True, "second one") + + network.add_obj("__ret", ret1) + network.add_obj("__ret", ret2) + network.add_obj("__ret", ret3) + assert network.working_memory == { + WME("f-00000", "status", True), + WME("f-00000", "value", "first one"), + WME("f-00001", "status", False), + WME("f-00001", "value", "to discard"), + WME("f-00002", "status", True), + WME("f-00002", "value", "second one"), + } + + # Check that the rule is fired + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + + @pytest.mark.parametrize("conditions, expected", [ + ([Condition("id", "some_weird_attribute", "value")], {'id': ['some_weird_attribute']}), + ([Condition("id", "attr1", V('x')), Condition("id", "attr2", "*")], {'id': ['attr1', 'attr2']}), + ([Condition(V('x'), "attr", "value")], {}), + ([Condition("id", V('x'), "value")], {'id': ["*"]}), + ]) + def test_attributes_by_identifier_are_updated_upon_condition_creation(self, conditions, expected): + network = ReteNetwork() + + rule = RuleForTestingRete(AndConditions(conditions)) + network.add_rule(rule) + + assert network.attributes_by_id == expected + + def test_i_cannot_add_the_same_object_twice(self): + network = ReteNetwork() + + rule = RuleForTestingRete(AndConditions([Condition("__obj", "attr", "value")])) + network.add_rule(rule) + + foo = Concept("foo") + network.add_obj("__obj", foo) + + with pytest.raises(ValueError): + network.add_obj("__obj", foo) + + def test_i_can_remove_wme(self): + network = ReteNetwork() + + cond1 = Condition("id", "attr", "value") + wme = WME("id", "attr", "value") + rule = RuleForTestingRete(AndConditions([cond1])) + network.add_rule(rule) + network.add_wme(wme) + assert len(list(network.matches)) == 1 + + network.remove_wme(wme) + assert len(list(network.matches)) == 0 + + def test_i_can_remove_wme_when_multiple_p_nodes(self): + network = ReteNetwork() + + cond1 = Condition("id", "attr", V("x")) + cond2 = Condition("id", "attr", V("y")) + + rule1 = RuleForTestingRete(AndConditions([cond1])) + rule2 = RuleForTestingRete(AndConditions([cond2])) + network.add_rule(rule1) + network.add_rule(rule2) + + assert len(network.pnodes) == 2 + + wme = WME("id", "attr", "value") + network.add_wme(wme) + assert len(list(network.matches)) == 2 + + network.remove_wme(wme) + assert len(list(network.matches)) == 0 + + def test_i_can_remove_and_add_again_a_wme(self): + network = ReteNetwork() + + cond1 = Condition("id", "attr", "value") + wme = WME("id", "attr", "value") + rule = RuleForTestingRete(AndConditions([cond1])) + network.add_rule(rule) + network.add_wme(wme) + network.remove_wme(wme) + assert len(list(network.matches)) == 0 + + network.add_wme(wme) + assert len(list(network.matches)) == 1 + + def test_i_can_remove_a_fact(self): + sheerka, context, boy, the, greetings = self.init_concepts( + "boy", + Concept("the x").def_var("x"), + Concept("greetings", definition="hello a", definition_type=DEFINITION_TYPE_DEF).def_var("a") + ) + network = ReteNetwork() + + the_boy = sheerka.new(the, x=boy) + hello_the_boy = sheerka.new(greetings, a=the_boy) + + # to allow all attributes for '__ret' + rule = RuleForTestingRete(AndConditions([Condition(V("ret"), "__name__", "__ret"), + Condition(V("ret"), V("x"), V("y"))])) + network.add_rule(rule) + + ret = ReturnValueConcept("test", True, hello_the_boy) + network.add_obj("__ret", ret) + + # sanity check + assert len(network.working_memory) > 0 + assert len(list(network.matches)) > 0 + + # remove the object + network.remove_obj(ret) + + assert len(network.working_memory) == 0 + assert len(list(network.matches)) == 0 + + def test_i_cannot_remove_an_object_with_no_fact_id(self): + network = ReteNetwork() + with pytest.raises(ValueError): + network.remove_obj(object()) + + def test_i_cannot_remove_an_object_that_is_not_in_the_system(self): + network = ReteNetwork() + + obj = ReturnValueConcept("who", True, "value") + setattr(obj, FACT_ID, "xxx") + with pytest.raises(ValueError): + network.remove_obj(obj) + + def test_i_can_find_stack_of_two_blocks_to_the_left_of_the_red_block(self): + network = ReteNetwork() + c1 = Condition(V("x"), "on", V("y")) + c2 = Condition(V("y"), "left-of", V("z")) + c3 = Condition(V("z"), "color", "red") + rule = RuleForTestingRete(AndConditions([c1, c2, c3])) + network.add_rule(rule) + + wme1 = WME("B1", "on", "B2") + wme2 = WME("B1", "on", "B3") + wme3 = WME("B1", "color", "red") + wme4 = WME("B2", "on", "table") + wme5 = WME("B2", "left-of", "B3") + wme6 = WME("B2", "color", "blue") + wme7 = WME("B3", "left-of", "B4") + wme8 = WME("B3", "on", "table") + wme9 = WME("B3", "color", "red") + + network.add_wme(wme1) + network.add_wme(wme2) + network.add_wme(wme3) + network.add_wme(wme4) + network.add_wme(wme5) + network.add_wme(wme6) + network.add_wme(wme7) + network.add_wme(wme8) + network.add_wme(wme9) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + assert matches[0].token.wmes == [wme1, wme5, wme9] + + am1 = network.build_or_share_alpha_memory(c1) + am2 = network.build_or_share_alpha_memory(c2) + am3 = network.build_or_share_alpha_memory(c3) + assert am1.items == [wme1, wme2, wme4, wme8] + assert am2.items == [wme5, wme7] + assert am3.items == [wme3, wme9] + + dummy_join = am1.successors[0] + join_on_value_y = am2.successors[0] + join_on_value_z = am3.successors[0] + match_c0 = dummy_join.children[0] + match_c0c1 = join_on_value_y.children[0] + match_c0c1c2 = join_on_value_z.children[0] + assert [t.wmes for t in match_c0.items] == [[wme1], [wme2], [wme4], [wme8]] + assert [t.wmes for t in match_c0c1.items] == [[wme1, wme5], [wme2, wme7]] + assert [t.wmes for t in match_c0c1c2.items] == [[wme1, wme5, wme9]] + + t0 = ReteToken(ReteToken(None, None), wme1) + t1 = ReteToken(t0, wme5) + t2 = ReteToken(t1, wme9) + assert match_c0c1c2.items[0] == t2 + + def test_i_can_find_stack_of_two_blocks_to_the_left_of_one_that_is_not_red(self): + network = ReteNetwork() + c1 = Condition(V("x"), "on", V("y")) + c2 = Condition(V("y"), "left-of", V("z")) + c3 = NegatedCondition(V("z"), "color", "red") + rule = RuleForTestingRete(AndConditions([c1, c2, c3])) + network.add_rule(rule) + + wme1 = WME("B1", "on", "B2") + wme2 = WME("B1", "on", "B3") + wme3 = WME("B1", "color", "red") + wme4 = WME("B2", "on", "table") + wme5 = WME("B2", "left-of", "B3") + wme6 = WME("B2", "color", "blue") + wme7 = WME("B3", "left-of", "B4") + wme8 = WME("B3", "on", "table") + wme9 = WME("B3", "color", "red") + + network.add_wme(wme1) + network.add_wme(wme2) + network.add_wme(wme3) + network.add_wme(wme4) + network.add_wme(wme5) + network.add_wme(wme6) + network.add_wme(wme7) + network.add_wme(wme8) + network.add_wme(wme9) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule] + assert matches[0].token.wmes == [wme2, wme7, None] + + am1 = network.build_or_share_alpha_memory(c1) + am2 = network.build_or_share_alpha_memory(c2) + am3 = network.build_or_share_alpha_memory(c3) + assert am1.items == [wme1, wme2, wme4, wme8] + assert am2.items == [wme5, wme7] + assert am3.items == [wme3, wme9] # same than positif condition + + dummy_join = am1.successors[0] + join_on_value_y = am2.successors[0] + join_on_value_z = am3.successors[0] + match_c0 = dummy_join.children[0] + match_c0c1 = join_on_value_y.children[0] + match_c0c1c2 = join_on_value_z.children[0] + + assert [t.wmes for t in match_c0.items] == [[wme1], [wme2], [wme4], [wme8]] + assert [t.wmes for t in match_c0c1.items] == [[wme1, wme5], [wme2, wme7]] + assert type(join_on_value_z) == NegativeNode + assert [t.wmes for t in match_c0c1c2.items] == [[wme2, wme7, None]] + + def test_i_can_remove_a_rule(self): + network = ReteNetwork() + rule1 = RuleForTestingRete(AndConditions([Condition("id", "attr", "value")])) + + network.add_rule(rule1) + network.remove_rule(rule1) + + assert len(network.pnodes) == 0 + assert len(network.alpha_hash) == 0 + assert len(network.rules) == 0 + # assert len(network.attributes_by_id) == 0 + # assert len(network.default_attributes) == 0 + + def test_i_can_remove_a_rule_when_it_shares_p_node(self): + network = ReteNetwork() + rule1 = RuleForTestingRete(AndConditions([Condition("id", "attr", "value")]), name="rule1") + rule2 = RuleForTestingRete(AndConditions([Condition("id", "attr", "value")]), name="rule2") + + network.add_rule(rule1) + network.add_rule(rule2) + + network.remove_rule(rule1) + + assert len(network.pnodes) == 1 + assert network.pnodes[0].rules == [rule2] + assert network.rules == {rule2} + + # sanity + wme = WME("id", "attr", "value") + network.add_wme(wme) + + matches = list(network.matches) + assert len(matches) == 1 + assert matches[0].pnode.rules == [rule2] + assert matches[0].token.wmes == [wme] + + @pytest.mark.skip("Need a better attribute management") + def test_i_can_manage_attributes_by_key_on_rule_removal(self): + network = ReteNetwork() + rule1 = RuleForTestingRete(AndConditions([Condition("id", "attr", "value")])) + rule2 = RuleForTestingRete(AndConditions([Condition("id", "attr", V("x"))])) + rule3 = RuleForTestingRete(AndConditions([Condition("id", "attr2", V("x"))])) + + network.add_rule(rule1) + network.add_rule(rule2) + network.add_rule(rule3) + + # sanity + assert network.attributes_by_id == {} + + network.add_rule(rule1) + assert network.attributes_by_id == {} + + network.add_rule(rule3) + assert network.attributes_by_id == {} + + network.add_rule(rule2) + assert network.attributes_by_id == {} + + @pytest.mark.skip("Need a better attribute management") + def test_i_can_manage_default_attributes_on_rule_removal(self): + network = ReteNetwork() + rule1 = RuleForTestingRete(AndConditions([Condition("id", V("x"), "value")])) + rule2 = RuleForTestingRete(AndConditions([Condition("id", V("y"), V("z"))])) + rule3 = RuleForTestingRete(AndConditions([Condition("id2", V("x"), V("y"))])) + + network.add_rule(rule1) + network.add_rule(rule2) + network.add_rule(rule3) + + # sanity + assert network.default_attributes == set() + + network.add_rule(rule1) + assert network.default_attributes == set() + + network.add_rule(rule3) + assert network.default_attributes == set() + + network.add_rule(rule2) + assert network.default_attributes == set() + + def test_i_can_add_an_empty_rule(self): + network = ReteNetwork() + + rule = RuleForTestingRete(AndConditions([])) + network.add_rule(rule) + assert len(network.beta_root.children) == 1 + + network.remove_rule(rule) + assert len(network.beta_root.children) == 0 + + def test_i_can_match_when_duplicate_conditions(self): + network = ReteNetwork() + c1 = Condition(V('x'), 'self', V('y')) + c2 = Condition(V('x'), 'color', 'red') + c3 = Condition(V('x'), 'color', 'red') + + rule = RuleForTestingRete(AndConditions([c1, c2, c3])) + network.add_rule(rule) + + wme1 = WME('B1', 'self', 'B1') + wme2 = WME('B1', 'color', 'red') + network.add_wme(wme1) + network.add_wme(wme2) + + assert len(list(network.matches)) == 1 + + def test_i_can_match_when_duplicate_conditions2(self): + network = ReteNetwork() + c1 = Condition(V('x'), 'self', V('y')) + c2 = Condition(V('x'), 'color', 'red') + c3 = Condition(V('y'), 'color', 'red') + + rule = RuleForTestingRete(AndConditions([c1, c2, c3])) + network.add_rule(rule) + + wme1 = WME('B1', 'self', 'B1') + wme2 = WME('B1', 'color', 'red') + network.add_wme(wme1) + network.add_wme(wme2) + + assert len(list(network.matches)) == 1 + + def test_multiple_rules(self): + network = ReteNetwork() + c1 = Condition(V('x'), 'on', V('y')) + c2 = Condition(V('y'), 'left-of', V('z')) + c3 = Condition(V('z'), 'color', 'red') + c4 = Condition(V('z'), 'on', 'table') + c5 = Condition(V('z'), 'left-of', 'B4') + + rule1 = RuleForTestingRete(AndConditions([c1, c2, c3])) + rule2 = RuleForTestingRete(AndConditions([c1, c2, c4, c5])) + + network.add_rule(rule1) + network.add_rule(rule2) + + wmes = [ + WME('B1', 'on', 'B2'), + WME('B1', 'on', 'B3'), + WME('B1', 'color', 'red'), + WME('B2', 'on', 'table'), + WME('B2', 'left-of', 'B3'), + WME('B2', 'color', 'blue'), + WME('B3', 'left-of', 'B4'), + WME('B3', 'on', 'table'), + WME('B3', 'color', 'red'), + ] + for wme in wmes: + network.add_wme(wme) + + # add rule on the fly + rule3 = RuleForTestingRete(AndConditions([c1, c2, c4, c3])) + network.add_rule(rule3) + + p_node1 = rule1.rete_p_nodes[0] + p_node2 = rule2.rete_p_nodes[0] + p_node3 = rule3.rete_p_nodes[0] + + assert len(list(p_node1.activations)) == 1 + assert len(list(p_node2.activations)) == 1 + assert len(list(p_node3.activations)) == 1 + assert list(p_node1.activations)[0].wmes == [wmes[0], wmes[4], wmes[8]] + assert list(p_node2.activations)[0].wmes == [wmes[0], wmes[4], wmes[7], wmes[6]] + assert list(p_node3.activations)[0].wmes == [wmes[0], wmes[4], wmes[7], wmes[8]] + + network.remove_rule(rule3) + assert len(list(p_node3.activations)) == 0 + + def test_rule_is_added_to_rete_network_when_it_is_created(self): + sheerka, context, rule = self.init_test().with_exec_rules(("rule_name", "id.attr == 'value'", 'True')).unpack() + evaluation_service = sheerka.services[SheerkaEvaluateRules.NAME] + rete_network = evaluation_service.network + + assert len(rete_network.pnodes) == 1 + assert rete_network.pnodes[0].rules[0] == rule + + def test_rule_is_removed_to_rete_network_when_it_is_deleted(self): + sheerka, context, rule = self.init_test().with_exec_rules(("id.attr == 'value'", 'True')).unpack() + evaluation_service = sheerka.services[SheerkaEvaluateRules.NAME] + rete_network = evaluation_service.network + + sheerka.remove_rule(context, rule) + assert len(rete_network.pnodes) == 0 + + def test_rules_are_removed_upon_ontology_removal(self): + sheerka, context = self.init_test().unpack() + service = sheerka.services[SheerkaRuleManager.NAME] + evaluation_service = sheerka.services[SheerkaEvaluateRules.NAME] + rete_network = evaluation_service.network + + sheerka.push_ontology(context, "Testing Rule removal 1") + rule = Rule(ACTION_TYPE_EXEC, "name", "id.attr = 'value'", "True") + rule = service.init_rule(context, rule) + service.create_new_rule(context, rule) + + assert len(rete_network.pnodes) == 1 + assert rete_network.pnodes[0].rules[0] == rule + + sheerka.push_ontology(context, "Testing Rule removal 2") + rule2 = Rule(ACTION_TYPE_EXEC, "name2", "id2.attr2 = 'value2'", "True") + rule2 = service.init_rule(context, rule2) + service.create_new_rule(context, rule2) + + assert len(rete_network.pnodes) == 2 + assert rete_network.pnodes[0].rules[0] == rule + assert rete_network.pnodes[1].rules[0] == rule2 + + sheerka.pop_ontology(context) + assert len(rete_network.pnodes) == 1 + assert rete_network.pnodes[0].rules[0] == rule + + sheerka.pop_ontology(context) + assert len(rete_network.pnodes) == 0