No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Mind Change Speed-up for Learning Languages from Positive Data
Abstract
Within the frameworks of learning in the limit of indexed classes of recursive languages from positive data and automatic learning in the limit of indexed classes of regular languages (with automatically computable sets of indices), we study the problem of minimizing the maximum number of mind changes FM(n) by a learner M on all languages with indices not exceeding n. For inductive inference of recursive languages, we establish two conditions under which FM(n) can be made smaller than any recursive unbounded non-decreasing function. We also establish how FM(n) is affected if at least one of these two conditions does not hold. In the case of automatic learning, some partial results addressing speeding up the function F M(n) are obtained.
ResearchGate has not been able to resolve any citations for this publication.
Duplicate of:
https://www.researchgate.net/publication/316715194_On_speed-up_of_function_synthesis_and_prediction
A natural ωpLω+1 hierarchy of successively more general criteria of success for inductive inference machines is described based on the size of sets of anomalies in programs synthesized by such machines. These criteria are compared to others in the literature. Some of our results are interpreted as tradeoff results or as showing the inherent relative-computational complexity of certain processes and others are interpreted from a positivistic, mechanistic philosophical stance as theorems in philosophy of science. The techniques of recursive function theory are employed including ordinary and infinitary recursion theorems.
This survey includes principal results on complexity of inductive inference for recursively enumerable classes of total recursive functions. Inductive inference is a process to find an algorithm from sample computations. In the case when the given class of functions is recursively enumerable it is easy to define a natural complexity measure for the inductive inference, namely, the worst-case mindchange number for the first n functions in the given class. Surely, the complexity depends not only on the class, but also on the numbering, i.e. which function is the first, which one is the second, etc. It turns out that, if the result of inference is Goedel number, then complexity of inference may vary between log2n+o(log2n) and an arbitrarily slow recursive function. If the result of the inference is an index in the numbering of the recursively enumerable class, then the complexity may go up to const·n. Additionally, effects previously found in the Kolmogorov complexity theory are discovered in the complexity of inductive inference as well.
The time complexity of prediction strategies (the value f(m+1) is predicted from f(0),..., f(m)) is investigated. It turns out that, if a prediction strategy F is "error-optimal" (i.e. it makes at most log2n+O(log2log2n) errors on the n-th function of the class), then the time complexity of computation of F(<f(0),..., f(m)>) (i.e. a candidate for f(m+1)) may go up, in some sense, to 22cm.
Special attention is paid to inductive inference by probabilistic algorithms. It turns out that arbitrary recursively enumerable class of total recursive functions can be identified with ln n + o(log n) mind- changes in an arbitrary numbering of the class.
We show that for some classes of recursive languages, from the characteristic function of any L in an approximate decision procedure for L with no more than n+1 mistakes can be (uniformly effectively) inferred in the limit; whereas, in general, a grammar (generation procedure) with no more than n mistakes cannot; for some classes an infinite sequence of perfectly correct decision procedures can be inferred in the limit, but single grammars with finitely many mistakes cannot; and for some classes an infinite sequence of decision procedures each with no more than n+1 mistakes can be inferred, but an infinite sequence of grammars each with no more than n mistakes cannot. This is true even though decision procedures generally contain more information than grammars. We also consider inference of grammars for r.e. languages from arbitrary texts, i.e., enumerations of the languages. We show that for any class of languages , if some, machine, from arbitrary texts for any L in , can infer in the limit an approximate grammar for L with no more than 2·n mistakes, then some machine can infer in the limit, for each language in , an infinite sequence of grammars each with no more than n mistakes. This reduction from 2·n to n is best possible. From these and other results we obtain and compare several natural, inference hierarchies. Lastly we show that if we restrict ourselves to recursive texts, there is a machine which, for any r.e. language, infers in the limit an infinite sequence of grammars each with only finitely many mistakes. We employ recursion theoretic methods including infinitary and ordinary recursion theorems.
The number of steps required to compute a function depends, in general, on the type of computer that is used, on the choice of computer program, and on the input-output code. Nevertheless, the results obtained in this paper are so general as to be nearly independent of these considerations.
A function is exhibited that requires an enormous number of steps to be computed, yet has a “nearly quickest” program: Any other program for this function, no matter how ingeniously designed it may be, takes practically as many steps as this nearly quickest program.
A different function is exhibited with the property that no matter how fast a program may be for computing this function another program exists for computing the function very much faster.
We study the learnability of enumerable families of uniformly recursive languages in dependence on the number of allowed mind changes, i.e., with respect to a well-studied measure of efficiency. We distinguish between exact learnability ( has to be inferred w.r.t. ) and class preserving learning ( has to be inferred w.r.t. some suitable chosen enumeration of all the languages from ) as well as between learning from positive and from both, positive and negative data.
The measure of efficiency is applied to prove the superiority of class preserving learning algorithms over exact learning. We considerably improve results obtained previously and establish two infinite hierarchies. Furthermore, we separate exact and class preserving learning from positive data that avoids overgeneralization. Finally, language learning with a bounded number of mind changes is completely characterized in terms of recursively generable finite sets. These characterizations offer a new method to handle overgeneralizations and resolve an open question of Mukouchi (1992).
We study the learnability of indexed families L = (L j ) j2IN of uniformly recursive languages under certain monotonicity constraints. Thereby we distinguish between exact learnability (L has to be learnt with respect to the space L of hypotheses), class preserving learning (L has to be inferred with respect to some space G of hypotheses having the same range as L), and class comprising inference (L has to be learnt with respect to some space G of hypotheses that has a range comprising range(L)). In particular, it is proved that, whenever monotonicity requirements are involved, then exact learning is almost always weaker than class preserving inference which itself turns out to be almost always weaker than class comprising learning. Next, we provide additionally insight into the problem under what conditions, for example, exact and class preserving learning procedures are of equal power. Finally, we deal with the question what kind of languages has to be added to the space of hypo...
We study the effect of query order on computational power, and show that \pjk-the languages computable via a polynomial-time machine given one query to the jth level of the boolean hierarchy followed by one query to the kth level of the boolean hierarchy-equals \redttnp{j+2k-1} if j is even and k is odd, and equals \redttnp{j+2k} otherwise. Thus, unless the polynomial hierarchy collapses, it holds that for each : \pjk = \pkj \iff (j=k) \lor (j{is even} \land k=j+1). We extend our analysis to apply to more general query classes.
A new view of learning is presented. The basis of this view is a natural notion of reduction. We prove completeness and relative difficulty results. An infinite hierarchy of intrinsically more and more difficult to learn concepts is presented. Our results indicate that the complexity notion captured by our new notion of reduction differs dramatically from the traditional studies of the complexity of the algorithms performing learning tasks. 2 1 Introduction Traditional studies of inductive inference have focused on illuminating various strata of learnability based on varying the definition of learnability. The research following the Valiant's PAC model [Val84] and Angluin's teacher/learner model [Ang88] paid very careful attention to calculating the complexity of the learning algorithm. We present a new view of learning, based on the notion of reduction, that captures a different perspective on learning complexity than all prior studies.
In this book we attempt to develop learning theory in systematic fashion, presupposing only basic notions from set theory and the theory of computation. Throughout our exposition, definitions and theorems are illustrated by consideration of language acquisition. However, no serious application of the theory is described.
The book is divided into three parts. Part I advances a fundamental model of learning due essentially to Gold (1967). Basic notation, terminology, and theorems are there presented, to be relied on in all subsequent discussion. In part II these initial definitions are generalized and varied in dozens of ways, giving rise to a multitude of learning models and theorems. We attempt to impose some order on these results through a system of notation and classification. Part III explores diverse issues in learning theory that do not fit neatly into the classification offered in part II. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
A new view of learning is presented. The basis of this view is a natural notion of reduction. We prove completeness and relative difficulty results. An infinite hierarchy of intrinsically more and more difficult to learn concepts is presented. Our results indicate that the complexity notion captured by our new notion of reduction differs dramatically from the traditional studies of the complexity of the algorithms performing learning tasks.KeywordsRecursive FunctionInductive InferenceIdentification TypeRecursive OperatorAcceptable NumberingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Assume a finite alphabet of constant symbols and a disjoint infinite alphabet of variable symbols. A pattern is a non-null finite string of constant and variable symbols. The language of a pattern is all strings obtainable by substituting non-null strings of constant symbols for the variables of the pattern. A sample is a finite nonempty set of non-null strings of constant symbols. Given a sample S, a pattern p is descriptive of S provided the language of p contains S and does not properly contain the language of any other pattern that contains S. The computational problem of finding a pattern descriptive of a given sample is studied. The main result is a polynomial-time algorithm for the special case of patterns containing only one variable symbol (possibly occurring several times in the pattern). Several other results are proved concerning the class of languages generated by patterns and the problem of finding a descriptive pattern.
The notion of the complexity of performing an inductive inference is defined. Some examples of the tradeoffs between the complexity of performing an inference and the accuracy of the inferred result are presented. An axiomatization of the notion of the complexity of inductive inference is developed and several results are presented which both resemble and contrast with results obtainable for the axiomatic computational complexity of recursive functions.
Studying the learnability of classes of recursive functions has attracted considerable interest for at least four decades. Starting with Gold’s (1967) model of learning in the limit, many variations, modifications and extensions have been proposed. These models differ in some of the following: the mode of convergence, the requirements intermediate hypotheses have to fulfill, the set of allowed learning strategies, the source of information available to the learner during the learning process, the set of admissible hypothesis spaces, and the learning goals.A considerable amount of work done in this field has been devoted to the characterization of function classes that can be learned in a given model, the influence of natural, intuitive postulates on the resulting learning power, the incorporation of randomness into the learning process, the complexity of learning, among others.On the occasion of Rolf Wiehagen’s 60th birthday, the last four decades of research in that area are surveyed, with a special focus on Rolf Wiehagen’s work, which has made him one of the most influential scientists in the theory of learning recursive functions.
Language learnability has been investigated. This refers to the following situation: A class of possible languages is specified, together with a method of presenting information to the learner about an unknown language, which is to be chosen from the class. The question is now asked, “Is the information sufficient to determine which of the possible languages is the unknown language?” Many definitions of learnability are possible, but only the following is considered here: Time is quantized and has a finite starting time. At each time the learner receives a unit of information and is to make a guess as to the identity of the unknown language on the basis of the information received so far. This process continues forever. The class of languages will be considered learnable with respect to the specified method of information presentation if there is an algorithm that the learner can use to make his guesses, the algorithm having the following property: Given any language of the class, there is some finite time after which the guesses will all be the same and they will be correct.In this preliminary investigation, a language is taken to be a set of strings on some finite alphabet. The alphabet is the same for all languages of the class. Several variations of each of the following two basic methods of information presentation are investigated: A text for a language generates the strings of the language in any order such that every string of the language occurs at least once. An informant for a language tells whether a string is in the language, and chooses the strings in some order such that every string occurs at least once.It was found that the class of context-sensitive languages is learnable from an informant, but that not even the class of regular languages is learnable from a text.
We study the complexity of decision problems that can be solved by a polynomial-time Turing machine that makes a bounded number of queries to an NP oracle. Depending on whether we allow some queries to depend on the results of other queries, we obtain two (probably) different hierarchies. We present several results relating the bounded NP query hierarchies to each other and to the Boolean hierarchy. We also consider the similarly defined hierarchies of functions that can be computed by a polynomial-time Turing machine that makes a bounded number of queries to an NP oracle. We present relations among these two hierarchies and the Boolean hierarchy.
The present work initiates the study of the learnability of automatic indexable classes which are classes of regular languages
of a certain form. Angluin’s tell-tale condition characterizes when these classes are explanatorily learnable. Therefore,
the more interesting question is when learnability holds for learners with complexity bounds, formulated in the automata-theoretic
setting. The learners in question work iteratively, in some cases with an additional long-term memory, where the update function
of the learner mapping old hypothesis, old memory and current datum to new hypothesis and new memory is automatic. Furthermore,
the dependence of the learnability on the indexing is also investigated. This work brings together the fields of inductive
inference and automatic structures.
Synthesis of programs of recursive functions from input/output examples is investigated where the synthesis is effective in the sense that one is able to algorithmically determine when the correct program of the function under consideration has been found. Necessary and sufficient conditions are derived that the synthesis can be done if the admissible number of input/output examples is bounded.
In this paper, we study the complexity of sets formed by boolean operations (union, intersection, and complement) on NP sets. These are the sets accepted by trees of hardware with NP predicates as leaves, and together these form the boolean hierarchy.We present many results about the structure of the boolean hierarchy: separation and immunity results, natural complete languages, and structural asymmetries between complementary classes.We show that in some relativized worlds the boolean hierarchy is infinite, and that for every k there is a relativized world in which the boolean hierarchy extends exactly k levels. We prove natural languages, variations of VERTEX COVER, complete for the various levels of the boolean hierarchy. We show the following structural asymmetry: though no set in the boolean hierarchy is -immune, there is a relativized world in which the boolean hierarchy contains -immune sets.Thus, this paper explores the structural properties of the...
Recently, a new approach to the study of “intrinsic” complexity of learning has originated in the work of Freivalds, and has been investigated for identification in the limit of functions by Freivalds, Kinber, and Smith and for identification in the limit of languages by Jain and Sharma. Instead of concentrating on the complexity of the learning algorithm, this approach uses the notion of reducibility to investigate the complexity of the concept classes being learned. Three representative classes have been presented that classify learning problems of increasing difficulty.(a)
Classes that can be learned by machines that confirm their success (singleton languages).
(b)
Classes that can be learned by machines that cannot confirm their success but can provide an upper bound on the number of mind changes after inspecting an element of the language (pattern languages).
(c)
Classes that can be learned by machines that can neither confirm success nor can provide an upper bound on the number of mind changes (finite languages).
The present paper shows that there is an infinite hierarchy of language classes that represent learning problems of increasing difficulty. Language classes constituting this hierarchy are languages with bounded cardinality, and it can be shown that collections of languages that can be identified using bounded number of mind changes are reducible to the classes in this hierarchy. It is also shown that language classes in this hierarchy are incomparable, under the reductions introduced, to the collection of pattern languages.
The structure of intrinsic complexity is shown to be rich as any finite, acyclic, directed graph can be embedded in this structure. However, it is also established that this structure is not dense.
Learnability of automatic classes
- S Jain
- Q Luo
- F Stephan
S. Jain, Q. Luo, and F. Stephan. Learnability of automatic classes. In Language and
Automata Theory and Applications, 4th International Conference, LATA 2010, volume
6031 of LNCS, pages 321-332. Springer, 2010.
Automatic presentations of structures, in: Logical and Computational Complexity
- B Khoussainov
- A Nerode
B. Khoussainov, A. Nerode, Automatic presentations of structures, in: Logical and Computational Complexity, (International Workshop LCC 1994), in: Lecture Notes in Computer Science, vol. 960, Springer, 1995, pp. 367–392.
Automatic learning with feedback queries, in: Models of Computation in Context, Seveth Conference on Computability in
- John Case
- Sanjay Jain
- Shin Yuh
- Pavel Ong
- Frank Semukhin
- Stephan
John Case, Sanjay Jain, Yuh Shin Ong, Pavel Semukhin, Frank Stephan, Automatic learning with feedback queries, in: Models of Computation in Context, Seveth Conference on Computability in Europe (CiE), in: Lecture Notes in Computer Science, vol. 6735, Springer-Verlag, 2011, pp. 31–40.
Learning recursive functions: A survey Special Issue on Forty Years of Inductive Inference
- S Zeugmann
- Zilles
Zeugmann and S. Zilles. Learning recursive functions: A survey. Theoretical Computer Science A, 397(1–3):4–56, 2008. Special Issue on Forty Years of Inductive Inference.
Dedicated to the 60th Birthday of Rolf Wiehagen.
Limiting synthesis of τ numbers
- J Bārzdiņš
J. Bārzdiņš. Limiting synthesis of τ numbers. In Theory of Algorithms and Programs,
vol. 1, pages 112-116. Latvian State University, Riga, Latvia, 1974. In Russian.
Concerning synthesis and prediction of functions
- J Bārzdiņš
- E Kinber
- K Podnieks
J. Bārzdiņš, E. Kinber, and K. Podnieks. Concerning synthesis and prediction of functions.
In Theory of Algorithms and Programs, vol. 1, pages 117-128. Latvian State University,
Riga, Latvia, 1974. In Russian.
Language learning with bounded number of mind changes
- S Lange
- T Zeugmann
S. Lange and T. Zeugmann. Language learning with bounded number of mind changes. In
Proceedings of the Tenth Annual Symposium on Theoretical Aspects of Computer Science,
pages 682-691. Springer-Verlag, 1993. Lecture Notes Computer Science, 665.
Special Issue on Forty Years of Inductive Inference
- T Zeugmann
- S Zilles
T. Zeugmann and S. Zilles. Learning recursive functions: A survey. Theoretical Computer Science A, 397(1-3):4-56, 2008. Special Issue on Forty Years of Inductive Inference.
Dedicated to the 60th Birthday of Rolf Wiehagen.