VSPACE50PT NOINDENT HUGEBFSERIES PREFACE VSKIP 20PTCHAPTERPREFACESECTIONWHY THIS BOOKTHE PURPOSE OF THIS BOOK IS TO BRIDGE THE GAP BETWEENINTRODUCTORYLEVEL SIGNAL PROCESSING CLASSES AND THE MATHEMATICSPREVALENT IN CURRENT SIGNAL PROCESSING RESEARCH AND PRACTICE  THE GAPIS BRIDGED BY PROVIDING A UNIFIED EM APPLIED TREATMENT OFFUNDAMENTAL MATHEMATICS SEASONED WITH DEMONSTRATIONS USING SC  MATLAB  THIS BOOK INTENDED NOT ONLY FOR STUDENTS OF SIGNALPROCESSING STILL PURSUING THEIR FORMAL EDUCATION BUT ALSO FORPRACTICING ENGINEERS WHO NEED TO BE ABLE TO ACCESS THE SIGNALPROCESSING RESEARCH LITERATURE AND FOR RESEARCHERS LOOKING FOR APARTICULAR RESULT THAT THEY WANT TO APPLY  IT IS THUS INTENDED BOTHAS A BF TEXTBOOK AND AS A BF REFERENCE  THE THEORY AND PRACTICE OF SIGNAL PROCESSING CONTRIBUTE TO AND DRAWFROM A VARIETY OF DISCIPLINES AMONG THEM CONTROLS COMMUNICATIONSSYSTEM IDENTIFICATION INFORMATION THEORY ARTIFICIAL INTELLIGENCESPECTROSCOPY PATTERN RECOGNITION TOMOGRAPHY IMAGE ANALYSIS ANDDATA ACQUISITION  TO FULFILL ITS ROLE IN THESE DIVERSE AREAS SIGNALPROCESSING EMPLOYS A VARIETY OF MATHEMATICAL TOOLS INCLUDINGTRANSFORM THEORY PROBABILITY OPTIMIZATION DETECTION THEORYESTIMATION THEORY NUMERICAL ANALYSIS LINEAR ALGEBRA FUNCTIONALANALYSIS AND MANY OTHERS  THE PRACTITIONER OF SIGNAL PROCESSING THE SIGNAL PROCESSOR  MAY USE SEVERAL OF THESE TOOLS IN THESOLUTION OF A PROBLEM FOR EXAMPLE BY SETTING UP A SIGNALRECONSTRUCTION ALGORITHM AND THEN OPTIMIZING THE PARAMETERS OF THEALGORITHM FOR OPTIMUM PERFORMANCE  MOST PRACTICING SIGNAL PROCESSORSMUST HAVE KNOWLEDGE OF BOTH THE BF THEORY AND THE BF  IMPLEMENTATION OF THE MATHEMATICS HOW AND WHY IT WORKS AND HOW TOMAKE THE COMPUTER DO IT  THE BREADTH OF MATHEMATICS EMPLOYED INSIGNAL PROCESSING COUPLED WITH THE OPPORTUNITY TO APPLY THE MATH TOPROBLEMS OF ENGINEERING INTEREST MAKES THE FIELD BOTH INTERESTING ANDREWARDINGTHE MATHEMATICAL ASPECTS OF SIGNAL PROCESSING ALSO INTRODUCE SOME OFITS MAJOR CHALLENGES HOW IS A STUDENT OR ENGINEERING PRACTITIONER TOBECOME VERSED IN THE VARIETY OF MATHEMATICAL TECHNIQUES WHILE STILLKEEPING AN EYE TOWARD THE APPLICATIONS  INTRODUCTORY TEXTS ON SIGNALPROCESSING TEND TO FOCUS HEAVILY ON TRANSFORM TECHNIQUES ANDFILTERBASED APPLICATIONS  WHILE AN ESSENTIAL PART OF THE TRAINING OFA SIGNAL PROCESSOR THIS FOCUS REVEALS ONLY THE TIP OF THE ICEBERG OFMATERIAL REQUIRED BY A LIFELONG PRACTICING ENGINEER  MORE ADVANCEDTEXTS USUALLY DEVELOP THE MATHEMATICAL TOOLS SPECIFIC TO A NARROWASPECT OF SIGNAL PROCESSING WHILE PERHAPS MISSING CONNECTIONS OFTHESE IDEAS TO RELATED AREAS OF RESEARCH  NEITHER OF THESE APPROACHESPROVIDES THE BACKGROUND NECESSARY TO READ AND UNDERSTAND BROADLY INTHE SIGNAL PROCESSING RESEARCH LITERATURE NOR TO EQUIP A PERSON WITHMANY SIGNAL PROCESSING TOOLSOVER THE YEARS THE SIGNAL PROCESSING LITERATURE HAS MOVED TOWARDINCREASING SOPHISTICATION  AS EXAMPLES APPLICATIONS OF THE SINGULARVALUE DECOMPOSITION SVD OR WAVELET TRANSFORMS ABOUND EVERYONE KNOWSSOMETHING ABOUT THESE BY NOW OR SHOULD  PART OF THIS MOVE TOWARDSOPHISTICATION IS FUELED BY COMPUTERS SINCE COMPUTATIONS FORMERLYREQUIRING CONSIDERABLE EFFORT AND UNDERSTANDING ARE NOW EMBODIED INCONVENIENT MATHEMATICAL PACKAGES  NAIVELY VIEWED THIS AUTOMATIONTHREATENS THE EXPERTISE OF THE ENGINEER WHY HIRE SOMEONE TO DO WHATCAN BE DONE IN TEN MINUTES WITH A SC MATLAB TOOLBOX  VIEWED MOREPOSITIVELY THE POWER OF THE COMPUTER PROVIDES A VARIETY OF NEWOPPORTUNITIES AS ENGINEERS ARE FREED FROM COMPUTATIONAL DRUDGERY TOPURSUE NEW APPLICATIONS  COMPUTER SOFTWARE AVAILABLE NOW PROVIDESPLATFORMS UPON WHICH INNOVATIVE IDEAS MAY BE DEVELOPED WITH GREATEREASE THAN EVER BEFORE  TAKING ADVANTAGE OF THE NEW FREEDOM TO DEVELOPUSEFUL CONCEPTS WILL REQUIRE A SOLID UNDERSTANDING OF MATHEMATICSBOTH TO APPRECIATE WHAT IS IN THE TOOLBOXES AND TO EXTEND BEYOND THEMTHIS BOOK IS INTENDED TO PROVIDE A FOUNDATION TO THE REQUISITEMATHEMATICSONE WAY FOR ASPIRINGPRACTITIONERS OF SIGNAL PROCESSING TO GET THE MATHEMATICAL BACKGROUNDTHEY NEED IS SIMPLY TO TAKE MORE MATHEMATICS CLASSES  WHILERECOMMENDED AS AN IDEAL FOR MANY SUCH A PROGRAM IS IMPRACTICAL THEYMAY FIND A COURSE IN PURE MATH TOO FAR REMOVED FROM THEIR OR THEIREMPLOYERS NEED FOR PRACTICAL KNOWLEDGEBEGINQUOTESOURCEWENDELL BERRYEM RECOLLECTED ESSAYS 19651980    P 197WHAT IS IT GOOD FOR WE ASK  AND ONLY IF IT PROVESIMMEDIATELY TO BE GOOD EM FOR SOMETHING ARE WE READY TO RAISE THEQUESTION OF VALUE HOW MUCH IS IT WORTH  BUT WE MEAN HOW MUCH MONEYFOR IF IT CAN ONLY BE GOOD FOR SOMETHING ELSE THEN OBVIOUSLY IT CANONLY BE EM WORTH SOMETHING ELSE  EDUCATION BECOMES TRAINING ASSOON AS WE DEMAND IN THIS SPIRIT THAT IT SERVE SOME IMMEDIATEPURPOSE AND THAT IT BE WORTH A PREDETERMINED AMOUNT  ONCE WE ACCEPTSO SPECIFIC A NOTION OF UTILITY ALL LIFE BECOMES SUBSERVIENT TO ITSUSE ITS VALUE IS DRAINED INTO ITS USE  ENDQUOTESOURCEBEGINQUOTESOURCEPATRICK BILLINGSLEYPREFACE P V EMPROBABILITY AND MEASURE 1986EDWARD DAVENANT SAID HE WOULD HAVE A MAN KNOCKT IN THE HEAD THATSHOULD WRITE ANYTHING IN MATHEMATIQUES THAT HAD BEEN WRITTEN OFBEFORE  LDOTS WHAT IS NEW HERE THENENDQUOTESOURCEBEGINQUOTESOURCEHENRY DAVID THOREAU  FOR EVERY THOUSAND HACKING AT THE LEAVES OF EVIL THERE IS ONE  STRIKING AT THE ROOTENDQUOTESOURCETHE LEVEL OF THIS BOOK ASSUMES THAT STUDENTS HAVE HAD A COURSE INTRADITIONAL TRANSFORMBASED DSP AT THE SENIOR OR FIRSTYEAR GRADUATELEVEL AND ALSO A TRADITIONAL COURSE IN STOCHASTIC PROCESSES  WHILECONCEPTS IN THESE AREAS ARE REVIEWED THIS BOOK DOES NOT SUPPLANT THEMORE FOCUSED COVERAGE THAT THESE COURSES CAN PROVIDESECTIONFEATURES OF THE BOOKSOME HIGHLIGHTS OF THE BOOK INCLUDEBEGINITEMIZEITEM AN EMPHASIS ON VECTORSPACE GEOMETRY WHICH PUTS LEASTSQUARES  AND MINIMUM MEANSQUARES IN THE SAME FRAMEWORK  THE CONCEPT OF  SIGNALS AS VECTORS IN AN APPROPRIATE VECTOR SPACE IS EMPHASIZED  THE VECTOR SPACE APPROACH PROVIDES A NATURAL FRAMEWORK FOR TOPICS  SUCH AS WAVELET TRANSFORMS AND DIGITAL COMMUNICATIONS AS WELL AS  THE TRADITIONAL TOPICS SUCH AS OPTIMUM PREDICTION FILTERING AND  ESTIMATION  IN THIS CONTEXT THE MORE GENERAL NOTION OF METRIC  SPACES IS INTRODUCED WITH A DISCUSSION OF SIGNAL NORMSITEM A THOROUGH DESCRIPTION OF THE LINEAR ALGEBRA USED IN SIGNAL  PROCESSING BOTH IN CONCEPT AND IN NUMERICAL IMPLEMENTATION  WHILE  LIBRARIES ARE COMMONLY AVAILABLE TO DO LINEAR ALGEBRA COMPUTATIONS  WE FEEL THAT THE NUMERICAL TECHNIQUES PRESENTED EXERCISE INTUITION  ON THE GEOMETRY OF VECTOR SPACES AND BUILD UNDERSTANDING OF THE  ISSUES THAT MUST BE ADDRESSED IN PRACTICAL PROBLEMS  THE LINEAR ALGEBRA INCLUDES A THOROUGH DISCUSSION OF EIGENBASED  METHOD OF COMPUTATION INCLUDING EIGENFILTERS MUSIC AND ESPRIT  THERE IS ALSO A CHAPTER DEVOTED TO THE PROPERTIES AND APPLICATIONS  OF THE SVD  TOEPLITZ MATRICES WHICH APPEAR THROUGHOUT THE SIGNAL  PROCESSING LITERATURE ARE TREATED BOTH FROM A NUMERICAL POINT OF  VIEW  AS AN EXAMPLE OF RECURSIVE ALGORITHMS  AND ALSO IN  CONJUNCTION WITH THE LATTICEFILTERING INTERPRETATION      THE MATRICES IN LINEAR ALGEBRA ARE VIEWED AS OPERATORS AND THE  IMPORTANT CONCEPT OF AN OPERATOR IS INTRODUCED  ASSOCIATED NOTIONS  SUCH AS RANGE NULLSPACE AND NORM OF AN OPERATOR ARE PRESENTED  WHILE A FULL COVERAGE OF OPERATOR THEORY IS NOT PROVIDED THERE IS A  STRONG FOUNDATION HERE THAT SERVES TO BUILD INSIGHT FOR OTHER  OPERATORS  ITEM IN ADDITION TO THE LINEAR ALGEBRAIC CONCEPTS A DISCUSSION OF  EM COMPUTATION IS ALSO PRESENTED  ALGORITHMS FOR COMPUTING THE  COMMON FACTORIZATIONS EIGENVALUES EIGENVECTORS SVDS AND MANY  OTHERS ARE PRESENTED WITH SOME NUMERICAL CONSIDERATION FOR  IMPLEMENTATION  WHILE NOT ALL OF THIS MATERIAL IS NECESSARILY  INTENDED FOR CLASSROOM USE IN CONVENTIONAL SIGNAL PROCESSING CLASSES   THERE IS NOT TIME FOR ALL OF THIS IN MOST CLASSES  THE  MATERIAL PROVIDES AN IMPORTANT PERSPECTIVE TO PERSPECTIVE  PRACTITIONERS AND A STARTING POINT FOR IMPLEMENTATIONS ON OTHER  PLATFORMS  INSTRUCTORS MAY CHOOSE TO EMPHASIZE CERTAIN NUMERIC  CONCEPTS BECAUSE THEY HIGHLIGHT THE GEOMETRY OF VECTOR SPACES  ITEM THE CAUCHYSCHWARTZ INEQUALITY IS USED IN A VARIETY OF PLACES AS  AN OPTIMIZING PRINCIPLEITEM RLS AND LMS ADAPTIVE FILTERS ARE PRESENTED AS NATURAL OUTGROWTHS  OF MORE FUNDAMENTAL CONCEPTS MATRIX INVERSE UPDATES AND STEEPEST  DESCENT  NEURAL NETWORKS AND BLIND SOURCE SEPARATION ARE ALSO  PRESENTED AS AN APPLICATION OF STEEPEST DESCENTITEM SEVERAL CHAPTERS ARE DEVOTED TO ITERATIVE AND RECURSIVE METHODS  EMPLOYED IN SIGNAL PROCESSING  WHILE ITERATIVE METHODS ARE OF GREAT  THEORETICAL AND PRACTICAL SIGNIFICANCE NO OTHER SIGNAL PROCESSING  TEXTBOOK PROVIDES THIS BREADTH OF COVERAGE  METHODS PRESENTED  INCLUDE PROJECTION ON CONVEX SETS COMPOSITE MAPPING THE EM  ALGORITHM CONJUGATE GRADIENT AND METHODS OF MATRIX INVERSE  COMPUTATION USING ITERATIVE METHODSITEM DETECTION AND ESTIMATION ARE PRESENTED WITH SEVERAL  APPLICATIONS INCLUDING SPECTRUM ESTIMATION PHASE ESTIMATION AND  MULTIDIMENSIONAL DIGITAL COMMUNICATIONSITEM OPTIMIZATION IS A KEY CONCEPT ON SIGNAL PROCESSING AND EXAMPLES  OF OPTIMIZATION BOTH UNCONSTRAINED AND CONSTRAINED APPEAR  THROUGHOUT THE TEXT  A THEORETICAL JUSTIFICATION FOR LAGRANGE  MULTIPLIER METHODS AS WELL AS THEIR PHYSICAL INTERPRETATION ARE  EXPLICITLY SPELLED OUT IN A CHAPTER ON OPTIMIZATION  A SEPARATE  CHAPTER DISCUSSES LINEAR PROGRAMMING AND ITS APPLICATIONSITEM IN ADDITION OPTIMIZATION ON GRAPHS SHORTEST PATH PROBLEMS ARE  ALSO EXAMINED ALONG WITH A VARIETY OF APPLICATIONS IN  COMMUNICATIONS AND SIGNAL PROCESSINGITEM THE EM ALGORITHM IS PRESENTED HERE THE ONLY TREATMENT KNOWN IN  A SIGNAL PROCESSING TEXTBOOK  THIS POWERFUL ALGORITHM IS USED FOR  MANY OTHERWISE INTRACTABLE ESTIMATION AND LEARNING PROBLEMSENDITEMIZETHE PRESENTATION IS AT A MORE FORMAL LEVEL THAN HAS BECOME TRADITIONALIN MANY RECENT DSP BOOKS FOLLOWING A THEOREMPROOF FORMATTHROUGHOUT THE TEXT  AT THE SAME TIME IT IS LESS FORMAL THAN MANYMATH BOOKS COVERING THIS MATERIAL  IN THIS WE HAVE ATTEMPTED TO HELPTHE STUDENTS FEEL COMFORTABLE WITH RIGOROUS THINKING WITHOUTOVERWHELMING THE STUDENT WITH TECHNICALITIES  A BRIEF REVIEW OFMETHODS OF PROOFS IS ALSO PROVIDED TO HELP STUDENTS DEVELOP A SENSE OFHOW TO APPROACH PROOFS  ULTIMATELY THE AIM OF THE BOOK IS TOEDUCATE ITS READER IN HOW TO THINK ABOUT PROBLEMS  TO THIS END INSOME PLACES MATERIAL IS COVERED MORE THAN ONCE FROM DIFFERENTPERSPECTIVES EG MORE THAN ONE PROOF FOR SOME RESULTS TODEMONSTRATE THAT THERE IS USUALLY MORE THAN ONE WAY TO APPROACH APROBLEMTHROUGHOUT THE TEXT THE INTENT HAS BEEN TO EXPLAIN THE WHAT ANDWHY OF THE MATHEMATICS BUT WITHOUT BECOMING OVERWROUGHT WITH SOMEOF THE MORE TECHNICAL MATHEMATICAL OCCUPATIONS  IN THIS REGARD THEBOOK DOES NOT NECESSARILY THOROUGHLY TREAT QUESTIONS OF HOW WELLFOR EXAMPLE IN OUR COVERAGE OF LINEAR NUMERICAL ANALYSIS THEPERTURBATION ANALYSIS THAT CHARACTERIZES MUCH OF THE RESEARCHLITERATURE HAS BEEN LARGELY IGNORED  NOR DO ISSUES OF COMPUTATIONALCOMPLEXITY FORM A MAJOR CONSIDERATION  CONSIDER THIS AUTOMOTIVEANALOGY  OUR INTENT IS TO GET UNDER THE HOOD OF THE CAR TO ASUFFICIENT LEVEL THAT IT IS CLEAR WHY THE ENGINE RUNS AND WHAT IT CANDO BUT WITHOUT PROVIDING A MOLECULARLEVEL DESCRIPTION OF THEMETALLURGICAL STRUCTURE OF THE PISTON RINGS  SUCH FINEGRAINEDINVESTIGATIONS ARE A NECESSARY PART OF THE RESEARCH INTO FINETUNINGTHE PERFORMANCE OF THE ENGINE  OR THE ALGORITHM  BUT ARE NOTAPPROPRIATE FOR A READER LEARNING THE MECHANICSTHROUGHOUT THE BOOK AND IN THE APPENDICES THERE IS ALSO GREAT DEAL OFMATERIAL THAT WILL BE OF REFERENCE VALUE TO PRACTICING ENGINEERS  FOREXAMPLE THERE ARE FACTS REGARDING MATRIX RANK THE INVERTIBILITY OFMATRICES PROPERTIES OF HERMITIAN MATRICES PROPERTIES OF STRUCTUREDMATRICES PRESERVED UNDER MULTIPLICATION AND AN EXTENSIVE TABLE OFGRADIENTS  NOT ALL OF THIS MATERIAL IS NECESSARILY INTENDED FORCLASSROOM USE IN CONVENTIONAL SIGNAL PROCESSING CLASSES BEINGPROVIDED TO ENHANCE THE VALUE OF THE BOOK AS A REFERENCENEVERTHELESS WHERE SUCH REFERENCE MATERIAL IS PROVIDED IT IS USUALLYACCOMPANIED BY AN EXPLANATION OF THE DERIVATION SO THAT RELATED FACTSNOT LISTED MAY OFTEN BE DERIVED BY THE READER  THE INTENT ALWAYS ISTO EDUCATE AND EMPOWER THE READER NOT SIMPLY PROVIDE THE ANSWERBEGINQUOTESOURCEWENDELL BERRYRECOLLECTED ESSAYS 19651980    P X FINALLY I WOULD LIKE TO ALERT THE READER TO MY CONVICTION  THAT THIS IS A PIECE OF UNFINISHED BUSINESS AND THAT MORE TIME AND  WORK WILL REVEAL FURTHER NEED OF CORRECTIONENDQUOTESOURCE NEWLENGTHKNUTHLENGTH SETTOWIDTHKNUTHLENGTH EM VOLUME I FUNDAMENTAL ALGORITHMS PP VIIVIII BEGINQUOTESOURCEDONALD KNUTHPARBOXTKNUTHLENGTHEM THE ART       OF COMPUTER PROGRAMMING PAR     EM VOLUME I FUNDAMENTAL ALGORITHMS PP VIIVIII MY ORIGINAL GOAL WAS TO BRING READERS TO FRONTIERS OF KNOWLEDGE IN EVERY SUBJECT THAT WAS TREATED  BUT IT IS EXTREMELY DIFFICULT TO KEEP UP WITH A FIELD THAT IS ECONOMICALLY PROFITABLE DOTS THE SUBJECT HAS BECOME A VAST TAPESTRY OF TENS OF THOUSANDS OF SUBTLE RESULTS CONTRIBUTED BY TENS OF THOUSANDS OF PEOPLE ALL OVER THE WORLD THEREFORE MY NEW GOAL HAS BEEN TO CONCENTRATE ON CLASSIC TECHNIQUES LIKELY TO REMAIN IMPORTANT FOR MANY MORE DECADES AND TO DESCRIBE THEM AS WELL AS I CAN ENDQUOTESOURCEAS WITH KNUTHS BOOK WHILE THIS BOOK WILL NOT PROVIDE THE FINAL WORD IN ANY RESEARCH AREAWE HOPE THAT FOR MANY RESEARCH PATHS IT WILL AT LEAST PROVIDE A GOODFIRST STEP  THE CONTENTS OF THE BOOK HAVE BEEN SELECTED ACCORDING TOA VARIETY OF CRITERIA  THE PRIMARY SELECTION CRITERION IS WHETHERMATERIAL HAS BEEN OF USE OR INTEREST TO US IN OUR RESEARCH  QUESTIONSFROM STUDENTS AND THE NEED TO FIND A CLEAR EXPLANATION FOR THEM HAVELEAD TO INCLUSION OF OTHER MATERIAL  THE EXCEPTIONAL WRITINGS FOUNDIN OTHER TEXTBOOKS AND PAPERS HAS BEEN A FACTOR  SOME OF THE MATERIALHAS BEEN INCLUDED FOR ITS PRACTICALITY AND SOME FOR ITS OUTSTANDINGBEAUTYTHERE IS ONGOING DEBATE REGARDING THE TEACHING OF MATHEMATICS TOENGINEERS  RECENT PROPOSALS SUGGEST USING JUST IN TIMEMATHEMATICS PROVIDING THE MATHEMATICAL CONCEPT ONLY WHEN THE NEED FORIT ARISES IN THE SOLUTION OF ENGINEERING PROBLEMS  THIS APPROACH HASARISEN AS A RESPONSE TO THE CHARGE THAT MATHEMATICAL PEDAGOGY HAS BEENPRESENTED USING A JUST IN CASE APPROACH WELL TEACH YOU ALL THISSTUFF JUST IN CASE YOU EVER HAPPEN TO NEED IT  IN REALITY NEITHER OFTHESE APPROACHES ARE EITHER FULLY DESIRABLE OR ACHIEVABLE POTENTIALLYLACKING RIGOR AND DEPTH ON THE ONE HAND AND LACKING MOTIVATION ANDINSIGHT ON THE OTHER  AS AN ALTERNATIVE WE HOPE THAT THEPRESENTATION IN THIS BOOK IS JUSTIFIED SO THAT THE LEVEL OFMATHEMATICS IS SUITED TO ITS APPLICATION AND THE APPLICATIONS ARESEEN IN CONJUNCTION WITH THE CONCEPTSIN ADDITION TO ATTEMPTING TO PROVIDE THOROUGH EXPLANATIONS OF MANYCORE TOPICS WE ALSO ATTEMPT TO PLANT SOME SEEDS OF IDEAS  THESEINCLUDE SUCH TOPICS AS COMMUTATIVE DIAGRAMS INFINITE PRODUCTSINCIDENCE MATRICES INFORMATION THEORY AND GRAPH THEORY  WE HAVEALSO ATTEMPTED IN THE TO EXPLAIN SOME OF THE LIMITATIONS OF THEMETHODS AND TO PROVIDE REFERENCES TO ALTERNATIVE TECHNIQUES  ALSOSINCE A MATERIAL IS LEARNED BEST BY APPRECIATING ITS CREATOR WE HAVEPROVIDED A FEW HISTORICAL VIGNETTES  THESE HAVE BEEN DRAWN MOSTLYFROM CITEBOYER CITEOTHERMATHHIST AND CITEMATHUNIVTHE GOAL OF PROVIDING A THOROUGH COVERAGE OF THE CONCEPTS IS FAR FROMACHIEVED IN THIS VOLUME  ALONG THE WAY WE WERE FORCED TO JETTISONENTIRE PARTS OF THE BOOK IN THE INTEREST IN OBTAINING AN OSTENSIBLYPORTABLE BOOK  FOR THOSE WITH AN INTEREST IN NUMBER THEORYPOLYNOMIAL THEORY INTERPOLATION AND APPROXIMATION INTEGRALEQUATIONS OR A VARIETY OF OTHER TOPICS WE EXPRESS OUR REGRETSSECTIONTHE PROGRAMSTHROUGHOUT THE TEXT THERE ARE MANY ALGORITHMS WRITTEN IN SC MATLABTHESE EXAMPLES WILL ALLOW THE READER TO SEE HOW THE CONCEPTS DEVELOPEDIN THE TEXT MIGHT BE IMPLEMENTED ALLOW EASY EXPLORATION OF THECONCEPTS AND SOMETIMES THE LIMITATIONS OF THE THEORY AND SHOULDPROVIDE A USEFUL LIBRARY OF CORE FUNCTIONALITY FOR A VARIETY OF SIGNALPROCESSING RESEARCH  WITH THE THOROUGH THEORETICAL AND APPLIEDDISCUSSION SURROUNDING AN ALGORITHM THIS BOOK IS NOT SIMPLY A RECIPEBOOK BUT THE INGREDIENTS ARE PROVIDED TO STIR UP SOME INTERESTINGSTEWSIN MOST CASES THE ALGORITHMS ARE NOT TYPESET IN THE BOOK  INSTEADTHE ICON PARNOINDENT INCLUDEGRAPHICSPICON NOINDENTIS USED TO INDICATE THAT AN ALGORITHM IS TO BE FOUND ON THE INCLUDEDCDROM  IN SOME INSTANCES THE ALGORITHM CONSISTS OF SEVERAL RELATEDFILESIN THE INTEREST OF BREVITY TYPE CHECKING OF ARGUMENTS HAS NOT BEENINCORPORATED INTO THE FUNCTIONS  OTHERWISE THE CODE IS BELIEVED TOWORK AT LEAST TO PRODUCE THE EXAMPLES DESCRIBED IN THE BOOK BUTBUGFIXES AND IMPROVEMENTS ARE ALWAYS WELCOMEWE MAKE THE STANDARD DISCLAIMER OF WARRANTY BF WE MAKE NO  WARRANTY EXPRESS OR IMPLIED THAT THE PROGRAMS OR ALGORITHMS  PRESENTED IN THIS BOOK OR ITS ACCOMPANYING MEDIA ARE FREE OF  ERROR OR THAT THEY WILL MEET YOUR REQUIREMENTS FOR ANY PARTICULAR  APPLICATIONS  THEY SHOULD NOT BE RELIED UPON FOR SOLVING A PROBLEM  WHOSE INCORRECT SOLUTION COULD RESULT IN INJURY TO A PERSON OR LOSS  OF PROPERTY  ANY AND ALL USE OF THE PROGRAMS AND ALGORITHMS  ASSOCIATED WITH THIS BOOK IS AT YOUR OWN RISK  THE AUTHORS AND  PUBLISHER DISCLAIM ALL LIABILITY FOR DIRECT OR CONSEQUENTIAL DAMAGES  RESULTING FROM YOUR USE OF THE PROGRAMSYOU ARE FREE TO USE THE PROGRAMS OR ANY DERIVATIVE OF THEM FOR ANYSCIENTIFIC PURPOSE BUT PLEASE REFERENCE THIS BOOK  UPDATED VERSIONSOF THE PROGRAMS AND OTHER INFORMATION CAN BE FOUND AT THE WEBSITE VERBSOME WEBSITE ADDRESS PROBABLY WWWPRENHALLCOMMOON  ANNA SHOULD  BE LOOKING INTO THISSECTIONEXERCISESEXERCISES ARE FOUND AT THE END OF EACH CHAPTER  THESE EXERCISES ARELOOSELY DIVIDED INTO SECTIONS BUT IT MAY BE NECESSARY TO DRAW FROMMATERIAL IN OTHER SECTIONS OR EVEN OTHER CHAPTERS IN ORDER TO SOLVESOME OF THE PROBLEMS  THERE ARE RELATIVELY FEW MERELY NUMERICAL EXERCISES  WITH THECOMPUTER DOING AUTOMATED COMPUTATIONS IN MANY CASES SIMPLY RUNNINGTHE NUMBERS DOESNT SEEM TO PROVIDE INFORMATIVE EXERCISES  READERSARE ENCOURAGED OF COURSE TO PLAY AROUND WITH THE ALGORITHMS PROVIDEDTO GET A SENSE OF HOW THEY WORK  INSIGHT CAN FREQUENTLY GAINED ONSOME DIFFICULT PROBLEMS BY TRYING SEVERAL RELATED NUMERICAL EXAMPLESTHE INTENT OF THE EXERCISES IS TO ENGAGE TO READER IN THE DEVELOPMENTOF THE THEORY IN THE BOOK  MANY OF THE EXERCISES ARE TO PROVIDEDERIVATIONS FOR RESULTS PRESENTED IN THE CHAPTERS OR TO PROVE SOME OFTHE LEMMAS AND THEOREMS  OTHER EXERCISES REQUIRE PROGRAMMING ANEXTENSION OR MODIFICATION OF A SC MATLAB ALGORITHM PRESENTED IN THECHAPTER  STILL OTHER EXERCISES LEAD THE STUDENT THROUGH ASTEPBYSTEP PROCESS LEADING TO SOME SIGNIFICANT RESULTS FOR EXAMPLEA DERIVATION OF GAUSSIAN QUADRATURE A DERIVATION OF LINEAR PREDICTIONTHEORY EXTENSION OF INVERSES OF TOEPLITZ MATRICES OR ANOTHERDERIVATION OF THE KALMAN FILTER  WE HOPE THAT AS STUDENTS WORKTHROUGH THESE EXERCISES THEY WILL DEVELOP SKILL IN ORGANIZING THEIRTHINKING TO APPROACH OTHER PROBLEMS AS WELL AS ACQUIRE BACKGROUND INA VARIETY OF IMPORTANT TOPICSMOST OF THE EXERCISES REQUIRE A FAIR DEGREE OF INSIGHT AND EFFORT TOSOLVE  STUDENTS SHOULD PLAN ON BEING CHALLENGED  WHEREVER POSSIBLESTUDENTS SHOULD BE ENCOURAGED TO INTERACT WITH THE COMPUTER FORCOMPUTATIONAL ASSISTANCE INSIGHT AND FEEDBACKA SOLUTIONS MANUAL IS AVAILABLE TO INSTRUCTORS TO INSTRUCTORS WHO HAVEADOPTED THE BOOK FOR CLASSROOM USE  NOT ONLY ARE SOLUTIONS PROVIDEDBUT IN MANY CASES SC MATLAB AND SC MATHEMATICA CODE IS ALSOPROVIDED INDICATING HOW A PROBLEM MIGHT BE APPROACHED USING THECOMPUTER  BY PROVIDING GUIDANCE INTO HOW TO APPROACH THE PROBLEM THESOLUTIONS MANUAL CAN ALSO BE A VALUABLE RESOURCE FOR STUDENTS OFSIGNAL PROCESSING  SOLUTIONS TO SOME OF THE EXERCISES CAN BE FOUND ONTHE CDROM IE ANSWERS AT THE BACK OF THE BOOK THERE ARE SEVERAL DIFFERENT TYPES OF EXERCISES  SOME ARE MERELY COMPUTATIONAL  OTHERS INTRODUCE EXTENSIONS OF THE METHODS OF THE SECTION TO NEW PROBLEMS  IN SOME CASES THE EXERCISES ARE USED TO PRESENT NEW MATERIAL  OTHER EXERCISES REQUIRE THE STUDENT TO PROVE RESULTS USED IN THE TEXT WHILE OTHERS REQUIRE PROGRAM IMPLEMENTATION AND EVALUATION OF A CONCEPTSELECTION OF EXERCISES BY AN INSTRUCTOR CAN BE MADE ON THE BASIS OFTHE LEVEL OF PREPARATION OF THE STUDENTS AND THE AMOUNT OF TIME ASTUDENT IS EXPECTED TO SPEND WORKING PROBLEMSSECTIONPOSSIBLE COURSES OF STUDYTHERE IS SUFFICIENT MATERIAL HERE THAT A VARIETY OF USEFUL COURSESCOULD BE PUT TOGETHER USING THIS BOOK  THERE IS CLEARLY MOREINFORMATION IN THIS BOOK THAN CAN BE COVERED IN A SINGLE SEMESTER OREVEN A FULL YEAR  SEVERAL DIFFERENT COURSES OF STUDY COULD BE DEVISEDBASED ON THIS BOOK AND INSTRUCTORS ARE PROVIDED THE OPPORTUNITY TOCHOOSE THE MATERIAL SUITABLE FOR THE NEEDS AND DEVELOPMENT OF THEIRSTUDENTS  FOR EXAMPLE DEPENDING ON THE FOCUS OF THE CLASSINSTRUCTORS MAY CHOOSE TO SKIP COMPLETELY THE NUMERICAL ASPECTS OFALGORITHMS OR THEY MAY CHOOSE THEM AS A FOCUS OF THE COURSEHERE ARE SOME POSSIBLE COURSE OPTIONSBEGINENUMERATEITEM THE MATERIAL IN THE FIRST TWO PARTS IS REGARDED AS FOUNDATIONAL  UPON WHICH THE MAJOR CONCEPTS OF SIGNAL PROCESSING ARE BUILT  THE  FIRST PART PROVIDES A REVIEW OF SIGNAL MODELS AND REPRESENTATIONS  EG DIFFERENCE EQUATIONS TRANSFER FUNCTIONS STATE SPACE FORM  AND INTRODUCES SEVERAL IMPORTANT SIGNAL PROCESSING PROBLEMS SUCH AS  SPECTRUM ESTIMATION AND SYSTEM IDENTIFICATION  THE SECOND PART  PROVIDES A THOROUGH FOUNDATION IN LINEAR ALGEBRA WORKING FROM AN  UNDERGRADUATE LEVEL UP THROUGH SEVERAL APPLICATIONS  SELECTIONS  FROM THESE FIRST TWO PARTS WITH POSSIBLE ADDITIONS FROM THE FIRST  APPENDIX ON MATHEMATICAL FUNDAMENTALS WOULD MAKE A SOLID  SINGLESEMESTER COURSE FOR A COURSE TITLED SOMETHING LIKE  MATHEMATICAL METHODS FOR SIGNALS AND SYSTEMS A POSSIBLE COURSE  SEQUENCE FOR SUCH A COURSE MIGHT BE AS FOLLOWS  BEGINITEMIZE  ITEM MOVE FAIRLY QUICKLY THROUGH CHAPTER 1 12 WEEKS  SOME MAY    WISH TO ENTIRELY SKIP SECTIONS 18 AND 110 DEPENDING ON INTEREST  ITEM IN CHAPTER 2 MOVE QUICKLY TO THE VECTOR SPACE CONCEPTS THEN    FOCUS ON THE CONCEPT OF ORTHOGONALITY  FOR MANY CLASSES IT MAY    BE USEFUL TO SKIP THE MORE TECHNICAL SECTIONS ASSOCIATED WITH    INFINITEDIMENSIONAL VECTOR SPACES FOR EXAMPLE SECTIONS 212    213 AND 216  APPROX 2 WEEKS  ITEM SPEND TIME IN CHAPTER 3 ON LEASTSQUARES AND MINIMUM    MEANSQUARE FILTERING AND ESTIMATION CONCEPTS AND THE DUAL    APPROXIMATION PROBLEM SECTIONS 31314  23 WEEKS  THEN    DEPENDING ON INTEREST EXAMINE EITHER WAVELET TRANSFORMS OR    DIGITAL COMMUNICATIONS FROM THIS GEOMETRIC VIEWPOINT 1 WEEK  ITEM IN CHAPTER 4 FOCUS ON SECTIONS 41 THROUGH 45 TO GET THE    GEOMETRY OF THE OPERATORS THEN 49 FOR A RETURN TO THE    LEASTSQUARES IDEA AND 410 FOR PRACTICAL COMPUTATION ISSUES    THEN INTRODUCE THE RLS FILTER IN SECTION 411 AND VISIT    PARTITIONED MATRIX INVERSES IN SECTION 412  23 WEEKS  ITEM IN CHAPTER 5 FOCUS ON SECTIONS 52 AND 53  THE QR    FACTORIZATION IN PARTICULAR IS A FOUNDATION FOR MANY SIGNAL    PROCESSING ALGORITHMS  IF A NUMERIC IMPLEMENTATION VIEWPOINT IS    NOT OF INTEREST THEN MATERIAL AFTER SECTION 535 MAY BE OMITTED    23 WEEKS  ITEM SECTIONS 61  65 CONSTITUTE THE PRINCIPAL THEORY OF THE    CHAPTER  AFTER THESE SECTIONS HAVE BEEN COVERED APPLICATIONS    DRAWN FROM SECTIONS 67 THROUGH 612 WITH 68 AND 69 ARE    PROBABLY OF THE MOST INTEREST  IF A NUMERIC FOCUS IS DESIRED    SECTION 614 MAY BE COVERED 23 WEEKS  ITEM THE THEORY OF THE SVD IN SECTIONS 71 THROUGH 75 SHOULD BE    COVERED FOLLOWED BY A SUBSET OF APPLICATIONS FROM SECTIONS 76    THROUGH 79  23 WEEKS  ITEM TOPICS RELATED TO SPECIAL MATRICES WITH SPECIAL EMPHASIS ON    TOEPLITZ MATRICES CAN FILL THE REMAINING TIME   ENDITEMIZEITEM THE MATERIAL FROM CHAPTERS 10 THROUGH 14 WOULD FIT WELL INTO A  FIRST COURSE ON DETECTION AND ESTIMATION ESPECIALLY WHEN  SUPPLEMENTED BY SOME OF THE MATERIAL ON LINEAR ALGEBRA SUCH AS  EIGENDECOMPOSITIONS AND THE SINGULAR VALUE DECOMPOSITION  ITEM AN ALTERNATE WAY OF USING THE BOOK IS IN A ONESEMESTER TOOLS  COURSE WHICH SELECTS TOPICS FROM PARTS I II AND III  ASSUMING  FAMILIARITY WITH CONTINUOUSTIME AND DISCRETETIME SYSTEMS TOPICS  IN THIS COURSE COULD INCLUDE  BEGINENUMERATE  ITEM THE MULTIVARIATE GAUSSIAN DENSITY SECTION 17  1 WEEK  ITEM ESSENTIAL VECTOR SPACE NOTIONS SECTIONS 21 THROUGH 26    210 213 214 AND 215 2 WEEKS  ITEM APPLICATIONS OF VECTOR SPACE CONCEPTS EG LEASTSQUARES AND    MINIMUM MEANSQUARES FILTERING SECTIONS 31 32 34 38    THROUGH 312 3 WEEKS  ITEM MATRIX FACTORIZATIONS SECTIONS 52 AND 53 NO NUMERIC    DISCUSSION  1 WEEK  ITEM SINGULAR VALUE DECOMPOSITIONS SECTIONS 71 72 73 75    WITH SOME APPLICATIONS SUCH AS SECTION 76 2 WEEKS  ITEM INTRODUCTION TO DETECTION AND ESTIMATION SECTIONS 101 102    103 105 106 1 WEEK  ITEM DETECTION THEORY SECTIONS 111 THROUGH 116    3 WEEKS  ITEM ESTIMATION THEORY SECTIONS 121 122 124 125 126 2    WEEKS  ITEM KALMAN FILTERING SECTIONS 131 132 OR 133 1 WEEK  ENDENUMERATEITEM ANOTHER COURSE COULD BE ITERATIVE METHODS FOR SIGNAL  PROCESSING WHICH WOULD FOCUS ON CHAPTERS IN PART IV  THE COURSE  MATERIAL COULD WELL BE ACCOMPANIED BY A STUDENT RESEARCH PROJECTITEM ANOTHER COURSE COULD BE METHODS OF OPTIMIZATION FOR SIGNAL  PROCESSING WHICH WOULD FOCUS ON CHAPTERS IN PART VITEM YET ANOTHER ALTERNATIVE IS A WRAPUP COURSE FOR STUDENTS IN THE  SIGNAL AND SYSTEM AREA WHO ARE FAMILIAR WITH THEIR TOPIC AREAS AND  WISH TO SHARPEN THEIR ANALYTICAL SKILLS SOMEWHAT  THIS COURSE COULD  BE SIMILAR TO THE FIRST ONE MENTIONED WITH LESS TIME SPENT IN  CHAPTER 1 AND MORE TIME SPENT EXAMINING NUMERICAL IMPLEMENTATIONS  TOPICS FROM THE LAST PARTS OF THE BOOK COULD ALSO BE SELECTEDENDENUMERATESECTIONACKNOWLEDGEMENTSBEGINQUOTESOURCEISAAC NEWTONIF I HAVE SEEN FURTHER IT IS BY STANDING ON YE SHOULDERS OFGIANTSI DO NOT KNOW WHAT I MAY APPEAR TO THE WORLD BUT TO MYSELF I SEEM TOHAVE BEEN ONLY LIKE A BOY PLAYING ON THE SEASHORE AND DIVERTINGMYSELF IN NOW AND THEN FINDING A SMOOTHER PEBBLE OR A PRETTIER SHELLTHAN ORDINARY WHILST THE GREAT OCEAN OF TRUTH LAY UNDISCOVERED BEFORE MEENDQUOTESOURCEFOR PROVIDING A CHALLENGING AND STIMULATING ENVIRONMENT IN WHICH THEDEVELOPMENT OF THIS BOOK COULD OCCUR I OFFER MY APPRECIATION TO THELATE DR RICHARD HARRIS CHAIRMAN OF THE ELECTRICAL AND COMPUTERENGINEERING DEPARTMENT AT UTAH STATE UNIVERSITY WHO PASSED AWAYSUDDENLY AS THE BOOK WAS NEARING COMPLETION  FOR SUGGESTIONSCOMMENTS AND MUCHNEEDED CRITICISM THE COMMENTS OF MANY REVIEWERSARE APPRECIATED  PAUL BECKER AND HIS ERSTWHILE GROUP ATADDISONWESLEYLONGMAN HAS PROVIDED FRIENDLY ENCOURAGEMENT AND IT HASBEEN A PLEASURE WORKING WITH HIM  THE PRODUCTION STAFF AT INTERACTIVECOMPOSITION CORPORATION HAVE BEEN MONUMENTALLY PRODUCTIVE AND I THANKTHEM MAKING THIS ALL COME TOGETHERFOR STIMULATING AND BAFFLING CONVERSATIONS AND QUESTIONS I THANK MYSTUDENTS  I AM GRATEFUL FOR COMMENTS SUGGESTIONS ENCOURAGEMENTSAND ADVICE FROM FRIENDS AND COLLEAGUES WHO HAVE READ PORTIONS OF THISAND PROVIDED INPUT  THE PART ON DETECTION AND ESTIMATION THEORY COMES FROM WYNN STIRLINGAND I AM GRATEFUL AND HONORED THAT HIS NOTES CAN BE INCORPORATED INTOTHIS BOOK AND FOR THE OPPORTUNITY TO COLLABORATE WITH HIMDESPITE THE ASSISTANCE REVIEWS OVERSIGHT AND EDITING OF MANYPEOPLE I AM SURE THERE STILL LURK UNDETECTED ERRORS  THESE ARE MINEAND I DEEPLY REGRET THEM  IF YOU FIND ANY PLEASE LET ME KNOW SOTHEY CAN BE STAMPED OUTTO THOSE WHO HAVE PLAYED ON THE SHORES OF KNOWLEDGE AND FOUND SO MANYBRILLIANT SHELLS I EXTEND ENTHUSIASTIC APPRECIATION  I ALSO THANKTHOSE WHO BY THEIR WRITING AND INTERPRETATIONS BY THEIR TEACHING ANDDEDICATION HAVE EXTENDED MY VIEWS BY HELPING ME CLIMB UP TOWARD THESHOULDERS OF THE GIANTS  MY PARENTS HAVE INSTILLED IN ME THECURIOSITY AND WONDER ABOUT THE WORLD AROUND ME  TO MY MOTHER THANKSFOR AN INSATIABLE CURIOSITY ABOUT LIFE  TO MY FATHER THANKS FORPROVIDING THE PATTERNMY MOST HEARTFELT THANKS GO TO BARBARA WHO MORE THAN ANYONE HASSHOULDERED WITH ME THE BURDEN OF SEEING THIS THROUGH AND HAS SHARED MEWITH THIS BOOK  SHE ALSO APPRECIATES THE NEED TO KNOW  THANKS ALSOTO OUR CHILDREN  LESLIE KYRA KAYLIE JENNIE KIANA AND SPENCER WHO PROVIDE MORE THAN SUFFICIENT REASON FOR JOY IN MY LIFE 1EMHFILL  TKM LOCAL VARIABLES TEXMASTER TEST END INTRODUCTORY CHAPTERBEGINTABBING MMQUAD  MMQUAD  MMQUAD  MMQUAD  KILL WHY THIS BOOK    MATHEMATICAL AREAS ENCOMPASSED AND EXAMPLE APPLICATIONS  TO BUILD INSIGHT AND MATURITY  TO PROVIDE A WINDOW ON SIGNAL PROCESSING LITERATURE MATHEMATICAL TOPICS ENCOMPASSED BY DSP OUTLINE AND STRUCTURE OF THE BOOK SOME CANONICAL PROBLEMS AND MODELS  THE MULTIVARIATE NORMAL MODEL   SYSTEM MODELING AND IDENTIFICATION  PREDICTION FILTERING AND SPECTRAL ESTIMATION  TRANSFER FUNCTION AND STATESPACE FORMS  STOCHASTIC MODELING HIDDEN MARKOV MODELS  SIGNAL DETECTION AND ESTIMATION MODAL ANALYSIS ARRAY PROCESSING  ESTIMATION KALMAN FILTERING  TIMEFREQUENCY ANALYSIS WAVELETSCHAPTERINTRODUCTIONLABELCHAPINTROBEGINQUOTESOURCEHUGH NIBLEYEM APPROACHING ZIONTHERE IS FULLTIME EMPLOYMENT FOR ALL SIMPLY IN EXPLORING THE WORLDWITHOUT DESTROYING IT AND BY THE TIME WE BEGIN TO UNDERSTANDSOMETHING OF ITS MARVELOUS RICHNESS AND COMPLEXITY WELL ALSO BEGINTO SEE THAT IT DOES HAVE USES WE NEVER SUSPECTEDLDOTSENDQUOTESOURCEBEGINQUOTESOURCEMICHAEL SPIVAKEM A COMPREHENSIVE INTRODUCTION TO    DIFFERENTIAL GEOMETRY  TODAY A DILEMMA CONFRONTS ANY ONE INTENT ON PENETRATING THE  MYSTERIES OF DIFFERENTIAL GEOMETRY  ON THE ONE HAND ONE CAN CONSULT  NUMEROUS CLASSICAL TREATMENTS OF THE SUBJECT IN AN ATTEMPT TO FORM  SOME IDEA HOW THE CONCEPTS WITHIN IT DEVELOPED  UNFORTUNATELY A  MODERN MATHEMATICAL EDUCATION TENDS TO MAKE CLASSICAL MATHEMATICAL  WORKS INACCESSIBLE LDOTS ON THE OTHER HAND ONE CAN NOW FIND TEXTS  AS MODERN IN SPIRIT AND CLEAN IN EXPOSITION AS BOURBAKIS  ALGEBRA  BUT A THOROUGH STUDY OF THESE BOOKS USUALLY LEAVES ONE  UNPREPARED TO CONSULT CLASSICAL WORKS AND ENTIRELY IGNORANT OF THE  RELATIONSHIP BETWEEN ELEGANT MODERN CONSTRUCTIONS AND THEIR CLASSICAL  COUNTERPARTS  MOST STUDENTS EVENTUALLY FIND THAT THIS IGNORANCE OF  THE ROOTS OF THE SUBJECT HAS ITS PRICE  NO ONE DENIES THAT MODERN  DEFINITIONS ARE CLEAR ELEGANT AND PRECISE  ITS JUST THAT ITS  IMPOSSIBLE TO COMPREHEND HOW ANY ONE EVER THOUGHT OF THEM  AND EVEN  AFTER ONE DOES MASTER A MODERN TREATMENT OF DIFFERENTIAL GEOMETRY  OTHER MODERN TREATMENTS OFTEN APPEAR SIMPLY TO BE ABOUT TO TOTALLY  DIFFERENT SUBJECTS LDOTS  AT THIS POINT I AM REMINDED OF A PAPER DESCRIBED IN LITTLEWOODS  EM MATHEMATICIANS MISCELLANY  THE PAPER BEGAN THE AIM OF THIS  PAPER IS TO PROVE LDOTS AND IT TRANSPIRED ONLY MUCH LATER THAT  THIS AIM WAS NOT ACHIEVED THE AUTHOR HADNT CLAIMED THAT IT WAS  WHAT I HAVE OUTLINED ABOVE IS THE CONTENT OF A BOOK THE REALIZATION  OF WHOSE PLAN AND THE INCORPORATION OF WHOSE DETAILS WOULD PERHAPS  BE IMPOSSIBLE WHAT I HAVE WRITTEN IS A SECOND OR THIRD DRAFT OF A  PRELIMINARY VERSION OF THIS BOOKENDQUOTESOURCESECTIONWHAT IS SIGNAL PROCESSINGTHE SCOPE OF SIGNAL PROCESSING FAR EXCEEDS THE CAPABILITY OF ANYSINGLE BOOK TO CONTAIN IT  THOUGH THE SUBJECT HAS GROWN SO BROAD ASTO OBVIATE A PERFECT AND PRECISE DEFINITION OF WHAT IS ENTAILED IN ITCERTAIN CONCEPTS MUST BE CONSIDERED AS INDISPENSABLE FOR RUDIMENTARYUNDERSTANDING  CERTAINLY SIGNAL PROCESSING INCLUDES THE MATERIALTAUGHT IN TRADITIONAL DIGITAL SIGNAL PROCESSING DSP COURSES SEEEG CITEPROAKIS1OPPENHEIMSCHAFER SUCH AS TRANSFORMS OF MANYVARIETIES Z LAPLACE FOURIER ETC AND THE CONCEPTS OF FREQUENCYRESPONSE IMPULSE RESPONSE AND CONVOLUTION FOR BOTH DETERMINISTICAND RANDOM SIGNALS  IT ALSO INCLUDES THE BASIC CONCEPTS OF FILTERINGAND FILTER DESIGN  THESE CONCEPTS ARE ASSUMED AS A BACKGROUND TO THISTEXT AND ARE USED AS NECESSARY THROUGHOUT THE TEXT  TRADITIONAL AREASIN SIGNAL PROCESSING INCLUDE AS TAKEN FROM THE IEEE EM TRANSACTIONS  ON SIGNAL PROCESSING CLASSIFICATIONS FILTER DESIGN FASTFILTERING ALGORITHMS TIMEFREQUENCY ANALYSIS MULTIRATE FILTERINGSIGNAL RECONSTRUCTION ADAPTIVE FILTERS NONLINEAR SIGNALS ANDSYSTEMS SPECTRAL ANALYSIS AND EXTENSIONS OF THESE CONCEPTS TOMULTIDIMENSIONAL SYSTEMS  THESE TOPICS ARE EMPLOYED IN A VARIETY OFAPPLICATION AREAS  IMPLEMENTATION IN HARDWARE OR SOFTWARE IS ALSOAN IMPORTANT FACET OF SIGNAL PROCESSING  PROVIDING A THOROUGHCOVERAGE OF THESE TOPICS ALONE REQUIRES MULTIPLE VOLUMESBUT IN THE VIEW OF THIS BOOK SIGNAL PROCESSING HAS AN EVEN GREATERREACH BECAUSE OF ITS INFLUENCE BY AND ON RELATED DISCIPLINES  SIGNALPROCESSING OVERLAPS WITH THE STUDY TRADITIONALLY KNOWN AS EM  CONTROLS SINCE CONTROL ULTIMATELY INVOLVES PRODUCING A SIGNALBASED UPON MEASURED OUTPUT OF A PLANT BY MEANS OF SOME PROCESSING UPONTHAT SIGNAL  BEFORE A SYSTEM CAN BE CONTROLLED THE PARTICULARPARAMETERS OF THAT SYSTEM USUALLY MUST BE DETERMINED SO EM SYSTEM  IDENTIFICATION IS AN ASPECT OF SIGNAL PROCESSING  THIS IN TURNRELATES TO EM SPECTRUM ESTIMATION AND ALL OF ITS APPLICATIONSSIGNAL PROCESSING HAS STRONG TIES TO EM COMMUNICATIONS THEORY ANDRECENTLY ESPECIALLY TO DIGITAL COMMUNICATION SINCE THE CAPABILITIESOF MODERN COMMUNICATION SYSTEMS ARE THE RESULT OF THE SIGNALPROCESSING PERFORMED WITHIN THEM  RELATED TO DIGITAL COMMUNICATIONARE QUESTIONS OF EM DETECTION AND EM ESTIMATION THEORY HOW TOGET THE BEST INFORMATION OUT OF SIGNALS MEASURED IN THE PRESENCE OFRANDOM NOISE  DETECTION AND ESTIMATION THEORY IN TURN RELATE TO EM  PATTERN RECOGNITION  DIGITAL COMMUNICATION ALSO SPILLS OVER INTOTHE AREAS OF EM INFORMATION THEORY AND EM CODING THEORY  SYSTEMIDENTIFICATION AND ESTIMATION THEORY TREAT QUESTIONS OF SOLVINGOVERDETERMINED SYSTEMS OF EQUATIONS THAT IN TURN HAVE APPLICATION INEM TOMOGRAPHY  THESE IN TURN HAVE SOME BEARING ON QUESTIONS OFAPPROXIMATION AND SMOOTHING OF SIGNALS  IF A TREATMENT OF FUNDAMENTALSIGNAL PROCESSING TOPICS REQUIRES SEVERAL VOLUMES THEN INCLUSION OFTHESE LATTER TOPICS REQUIRES A LIBRARYSIGNAL PROCESSING COVERS A LARGE TERRITORY  HOWEVER THERE IS ACOMMON THREAD AMONG ALL THE AREAS MENTIONED  THEY ALL INVOLVE AFAIR DEGREE OF MATHEMATICAL SOPHISTICATION AND IN BOTH THEORY ANDPRACTICE ASSUME AN ANALYTICAL AND A COMPUTATIONAL COMPONENT  MOST OFTHESE AREAS SHARE A LARGE OVERLAP IN CONCEPTUAL CONTENT  WEPROPOSE THE FOLLOWING AS A TENTATIVE DEFINITION OF SIGNAL PROCESSINGAT LEAST FOR THE PURPOSES OF THIS BOOKBEGINDEFINITION INDEXSIGNAL PROCESSING DEFINITION  BF SIGNAL PROCESSING IS THAT AREA OF APPLIED MATHEMATICS THAT DEALS  WITH OPERATIONS ON OR ANALYSIS OF SIGNALS IN EITHER DISCRETE OR  CONTINUOUS TIME TO PERFORM USEFUL OPERATIONS ON THOSE SIGNALSENDDEFINITIONWITH ITS FOCUS ON APPLIED MATHEMATICS THIS BOOK NEGLECTS SEVERALIMPORTANT ASPECTS OF SIGNAL PROCESSING INCLUDING HARDWARE DESIGN ANDIMPLEMENTATION ON SIGNAL PROCESSING CHIPS  USEFUL OPERATIONIS DELIBERATELY LEFT AMBIGUOUS  DEPENDING UPON THE APPLICATION AUSEFUL OPERATION COULD BE CONTROL DATA COMPRESSION DATATRANSMISSION DENOISING PREDICTION FILTERING SMOOTHING DEBLURRINGTOMOGRAPHIC RECONSTRUCTION IDENTIFICATION CLASSIFICATION OR AVARIETY OF OTHER OPERATIONS THE PRIMARY INTENT OF THIS BOOK IS BF TO PRESENT A TREATMENT OF  RELEVANT MATHEMATICS SO THAT STUDENTS AND PRACTITIONERS OF SIGNAL  PROCESSING AND RELATED FIELDS ARE ABLE TO READ APPLY AND  ULTIMATELY CONTRIBUTE TO A VARIETY OF AREAS OF SIGNAL PROCESSING  RESEARCH AND PRACTICE  THE INTENT IS NOT TO EXPLORE PUREMATHEMATICS HOWEVER BUT RATHER TO PROVIDE A MATHEMATICAL MODICUMSUFFICIENT TO EXPLAIN AND EXPLORE THE MORE IMPORTANT MATHEMATICALPARADIGMS USED IN SIGNAL PROCESSING EM ALGORITHMS  A STUDENT WITHA BACKGROUND FROM THIS BOOK SHOULD BE ABLE TO MOVE EXPEDITIOUSLY TO APARTICULAR AREA OF INTEREST AND BEGIN MAKING EFFECTIVE PROGRESS IN THESPECIALIZED LITERATURE OF THAT AREA  WE HAVE ENDEAVORED TO MAINTAIN APRECARIOUS BALANCE PURISTS IN MATHEMATICS WILL FIND SOME OF THEANALYTICAL METHODS DEFICIENT WHILE PRAGMATISTS WILL ARGUE THAT THEREARE FAR TOO MANY EQUATIONS  TO USE A GARAGE ANALOGY WE HAVE PROVIDEDENOUGH INFORMATION TO GET UNDER THE HOOD OF THE CAR TAKING APART FOREXAMINATION MANY OF THE ENGINE COMPONENTS BUT WITHOUT GETTING INTODETAIL AT THE LEVEL OF METALLURGICAL PHENOMENA  SUCH MINUTEINVESTIGATIONS ARE BEST CONDUCTED AFTER THE STUDENT UNDERSTANDS HOWTHE CAR OPERATES  IN ADDITION TO THEORY THE BOOK CONTAINS AVARIETY OF MATERIAL COMPARABLE TO WHAT IS FOUND IN OTHER ADVANCEDSIGNAL PROCESSING TEXTS  IN ADDITION TO THE PRIMARY GOAL OF THIS BOOK THERE ARE TWO OTHERSFIRST TO DEVELOP WITHIN THE STUDENT A DEGREE OF MATHEMATICALMATURITY  THE STUDENT WITH THIS MATURITY WILL IT IS HOPED BE ABLETO ORGANIZE EFFECTIVE APPROACHES OF HISHER OWN TO A VARIETY OFPROBLEMS  THIS MATURITY WILL BE DEVELOPED BY WORKING PROBLEMSFOLLOWING AND DOING PROOFS AND WRITING AND RUNNING PROGRAMSITEM IN ADDITION TO THE MATHEMATICAL CONTENT THE SC MATLAB  PROGRAMS PROVIDED IN THE BOOK SHOULD BE USEFUL BOTH AS STANDALONE  FUNCTIONS AND AS BUILDING BLOCKS TO FURTHER UNDERSTANDINGSECOND THE BOOK IS INTENDED AS A USEFUL REFERENCE WITH REFERENCEMATERIAL GATHERED ON SEVERAL AREAS IN SIGNAL PROCESSING SUCHAS DERIVATIVES LINEAR ALGEBRA OPTIMIZATION INEQUALITIES ETCTHIS STATEMENT OF INTENT SHOULD MAKE CLEAR WHAT THIS BOOK IS NOTTHERE ARE SEVERAL VERY GOOD BOOKS AVAILABLE ON APPLICATION AREAS INSIGNAL PROCESSING SUCH AS SPECTRUM ESTIMATION ADAPTIVE FILTERINGARRAY PROCESSING AND SO ON THIS BOOK DOES NOT CHOOSE ANY OF THOSEPARTICULAR AREAS AS ITS FOCUS  THUS WHILE MANY DIFFERENT TECHNIQUESOF SPECTRUM ESTIMATION WILL BE PRESENTED AS APPLICATIONS OF THETECHNIQUES DISCOVERED ISSUES CENTRAL TO THE STUDY OF SPECTRUMESTIMATION SUCH AS COMPARISONS OF THE DIFFERENT TECHNIQUES IN TERMSOF SPECTRAL RESOLUTION BIAS ETC ARE NOT PRESENTED HERESIMILARLY THE MAJOR PARADIGMS OF ADAPTIVE FILTERING ARE PRESENTED ASAPPLICATIONS OF OTHER IMPORTANT CONCEPTS EG LEASTSQUARES ANDMINIMUM MEANSQUARES AND RECURSIVE COMPUTATION OF MATRIX INVERSESBUT A THOROUGH TREATMENT OF THE CONVERGENCE OF THE FILTERS IS AVOIDEDRATHER THAN FOCUSING ON ONE PARTICULAR AREA OF RESEARCH INTEREST THISBOOK PRESENTS THE TOOLS THAT ARE USED IN THESE RESEARCH AREAS ENABLINGTHE INTERESTED STUDENT TO MOVE INTO A VARIETY OF DIFFERENT AREASSECTIONMATHEMATICAL TOPICS EMBRACED BY SIGNAL PROCESSINGSO WHAT DOES A SIGNAL PROCESSOR  THAT IS AN INDIVIDUAL WHO WANTSTO DESIGN SIGNAL PROCESSING ALGORITHMS NOT THE SPECIALIZEDMICROPROCESSOR THAT MIGHT BE USED TO IMPLEMENT THE ALGORITHMS  NEEDTO KNOW TO BE EFFECTIVE  DEPENDING ON THE PROBLEM SEVERALMATHEMATICAL TOOLS CAN BE EMPLOYEDBEGINDESCRIPTIONITEMLINEAR SIGNALS AND SYSTEMS AND TRANSFORM THEORY THESE TOPICS  CORE TO MANY UNDERGRADUATE AND INTRODUCTORY GRADUATE COURSES ARE  ASSUMED AS BACKGROUND TO THIS BOOK  FAMILIARITY WITH BOTH  CONTINUOUS AND DISCRETETIME SYSTEMS IS ASSUMED ALTHOUGH A REVIEW  OF SOME TOPICS IS PROVIDED IN SECTION REFSECLTI  ITEMPROBABILITY AND STOCHASTIC PROCESSES THIS IS A CRITICALLY  IMPORTANT AREA THAT IS ALSO ASSUMED AS BACKGROUND  STUDENTS SHOULD  BE ACQUAINTED WITH PROBABILITY AND HAVE HAD A COURSE IN STOCHASTIC  PROCESSES AS A PREREQUISITE TO THIS BOOK  PROBABILITY IS AN  IMPORTANT TOOL AND STUDENTS ARE ADVISED TO CONTINUE SHARPENING  THEIR SKILLS WITH IT  A BRIEF REVIEW OF IMPORTANT TOPICS IN  STOCHASTIC PROCESSES IS PROVIDED IN APPENDIX REFAPPDXRP  ITEMPROGRAMMING A SIGNAL PROCESSOR MUST KNOW HOW TO PROGRAM IN AT  LEAST ONE HIGHLEVEL LANGUAGE  IN MOST CASES SIGNAL PROCESSING  ULTIMATELY BOILS DOWN TO A SOFTWARE OR HARDWARE IMPLEMENTATION ON  SOME KIND OF COMPUTING PLATFORM  THIS REQUIRES DEPLOYMENT OF THE  CONCEPT SIMULATION AND TESTING ALL USUALLY SOFTWARERELATED  ACTIVITIES  AN UNDERSTANDING OF BASIC PROGRAMMING CONCEPTS SUCH AS  VARIABLES PROGRAM FLOW RECURSION DATA STRUCTURES AND PROGRAM  COMPLEXITY IS ASSUMEDITEMCALCULUS AND ANALYSIS THESE FOUNDATION CONCEPTS OCCUR  REPEATEDLY IN THE SIGNAL PROCESSING LITERATURE  A BROAD AND SHALLOW  COVERAGE OF ANALYSIS APPEARS IN APPENDIX REFAPPDXSETFUNCTITEMVECTOR SPACES AND LINEAR ALGEBRA WHILE EVERY UNDERGRADUATE  ENGINEER HAS SOME EXPOSURE TO LINEAR ALGEBRA THESE TOPICS ARE SO  IMPORTANT TO SIGNAL PROCESSING THAT ADDITIONAL EXPOSURE IS CRITICAL  MANY OF THE BASIC CONCEPTS ARE REVIEWED IN THIS BOOK WITH AN EYE  TOWARD APPLICATIONS IN SIGNAL PROCESSING  BECAUSE OF ITS  IMPORTANCE CHAPTERS REFCHAPVECTSP THROUGH REFCHAPKRONECKER  ARE DEVOTED LARGELY TO LINEAR ALGEBRA AND ITS APPLICATIONSITEMNUMERICAL METHODS WITH THE INCREASING PENETRATION OF COMPUTERS  INTO ENGINEERING CULTURE THERE IS PARADOXICALLY A DECREASE IN MANY  STUDENTS EXPOSURE TO NUMERICAL METHODS  AND YET A SIGNIFICANT  PORTION OF SIGNAL PROCESSING CONSISTS OF NOTHING MORE THAN NUMERICAL  METHODS APPLIED TO A PARTICULAR SET OF PROBLEMS INVOLVING SIGNALS  MANY OF THE TECHNIQUES DESCRIBED IN THIS BOOK ARE BORROWED FROM THE  NUMERICAL METHODS LITERATUREITEMFUNCTIONAL ANALYSIS IN SIGNAL PROCESSING A SIGNAL IS A  FUNCTION  THE TOOLS FROM FUNCTIONAL ANALYSIS PROVIDE A FRAMEWORK  FROM WHICH TO VIEW THE SIGNAL LEADING THE WAY TO POWERFUL SIGNAL  TRANSFORMS AND SIGNAL SPACES IN DIGITAL COMMUNICATIONS  IN THIS  BOOK WE PRESENT CONCEPTS FROM FUNCTIONAL ANALYSIS IN THE CONTEXT OF  VECTOR SPACES PARTICULARLY IN CHAPTERS REFCHAPVECTSP AND  REFCHAPVECTAPITEMSTATISTICAL DECISION THEORY STATISTICAL DECISION THEORY CAN BE  DESCRIBED AS THE SCIENCE OF MAKING DECISIONS IN THE FACE OF RANDOM  UNCERTAINTY  SUCH DECISIONMAKING ALSO DESCRIBES WHAT IS DONE IN  MANY SIGNAL PROCESSING APPLICATIONS  THE APPLICATION OF STATISTICS  TO SIGNAL PROCESSING CAN BE DIVIDED INTO TWO MAJOR OVERLAPPING  AREAS BF DETECTION THEORY AND BF ESTIMATION THEORY  DETECTION THEORY IS A FRAMEWORK FOR MAKING DECISIONS IN THE PRESENCE  OF NOISE  ESTIMATION THEORY PROVIDES A MEANS OF DETERMINING THE  VALUE OF A QUANTITY IN THE PRESENCE OF NOISE  DETECTION AND  ESTIMATION ARE COVERED IN CHAPTERS REFCHAPFORMALISM THROUGH  REFCHAPKALMANITEMOPTIMIZATION A COMMON THEME RUNNING THROUGH MANY SIGNAL  PROCESSING APPLICATIONS IS OPTIMIZATION WHATEVER IS BEING COMPUTED  WE WISH TO DO IT IN THE BEST POSSIBLE WAY  OR IF WE CANNOT GET TO  THE OPTIMAL OPERATION POINT IN ONE STEP WE WILL PROGRESS TOWARD IT  AS WE CONTINUE TO PROCESS DATA THAT IS WE WILL ADAPT  BECAUSE OF  ITS UBIQUITY IN APPLICATION IN PART REFPARTOPT WE PRESENT  FUNDAMENTAL CONCEPTS IN OPTIMIZATION INCLUDING AND CONSTRAINED  OPTIMIZATION LINEAR PROGRAMMING AND PATH SEARCH ALGORITHMS  IN  ADDITION OPTIMIZATION PROBLEMS PARTICULARLY FOR CONSTRAINED  OPTIMIZATION ARE PRESENTED THROUGHOUT THE TEXT AND IN THE  EXERCISESITEMMODERN ALGEBRA MODERN ALGEBRA PROVIDES A VOCABULARY OF  IMPORTANT CONCEPTS AND TOOLS USEFUL IN THE DEVELOPMENT OF SEVERAL  FAST ALGORITHMS  WE PRESENT THE BASIC DEFINITIONS AND SOME  USEFUL EXAMPLES IN SECTION REFSECALGEBRA WITH A FEW ADVANCED  EXAMPLES AND APPLICATIONS IN SECTION REFSECALG2ITEMCOMPLEX ANALYSIS ALL ENGINEERS KNOW ABOUT COMPLEX NUMBERS BUT  NOT ENOUGH KNOW ABOUT THE WONDERS OF COMPLEX ANALYSIS  SINCE  TRANSFORMS ALMOST INVARIABLY INVOLVE COMPLEX FUNCTIONS IT IS  IMPORTANT TO KNOW SOMETHING ABOUT COMPLEX ANALYSIS AND HOW IT  APPLIES TO TRANSFORM THEORY  THE ESSENTIALS ARE PRESENTED IN  CHAPTER REFCHAPCOMPLEXITEMPOLYNOMIAL THEORY POLYNOMIALS ARISE AS TRANSFER FUNCTIONS AND  CHARACTERISTIC EQUATIONS IN ANALYSIS OF LINEAR SYSTEMS  POLYNOMIALS  ARE ALSO DENSE IN THE SET OF CONTINUOUS FUNCTIONS WHICH MEANS THAT  THERE IS SOME POLYNOMIAL ARBITRARILY CLOSE TO ANY CONTINUOUS  FUNCTIONS  FOR MANY PRACTICAL PURPOSES WHATEVER WE WANT TO DO WITH  A CONTINUOUS FUNCTION WE CAN DO WITH A POLYNOMIAL  IN ADDITION  SEVERAL USEFUL ANALYTICAL HAVE BEEN DEVELOPED IN ASSOCIATION WITH  POLYNOMIALS SUCH AS THE ROUTHHURWITZ ALGORITHM AND THE JURY TEST  THESE AND OTHERS IMPORTANT CONCEPTS RELATED TO POLYNOMIALS ARE  PRESENTEDITEMNUMBER THEORY NUMBER THEORY THE STUDY OF INTEGERS AND THEIR  PROPERTIES ARISES IN SIGNAL PROCESSING BECAUSE NUMBERS REPRESENTED  IN A COMPUTER ARE ULTIMATELY INTEGERS  THE CONCEPTS OF NUMBER  THEORY PROVIDE A FRAMEWORK FOR SEVERAL FAST ALGORITHMS FOR  CONVOLUTION AND TRANSFORMS  SEE CHAPTER REFCHAPNUMTH FOR SOME  PRINCIPLES AND APPLICATIONSITEMAPPROXIMATION AND INTERPOLATION FILTER DESIGN IS FUNDAMENTALLY  AN EXERCISE IN APPROXIMATION CERTAIN FILTER REQUIREMENTS ARE KNOWN  AND IT IS DESIRED TO FIND A REALIZABLE FILTER THAT MEETS THE  REQUIREMENTS AS CLOSELY AS POSSIBLE  INTERPOLATION IS RELATED TO  UPSAMPLING HOW TO FIND OUT WHAT HAPPENS BETWEEN THE SAMPLES  CHAPTER REFCHAPINTERP DEVELOPS THESE IDEASITEMITERATIVE METHODS MANY SIGNAL PROCESSING METHODS CONVERGE TO  THEIR SOLUTION AFTER SEVERAL ITERATIONS  FOR EXAMPLE ADAPTIVE  FILTERS AND NEURAL NETWORKS  WE PRESENT SOME BASIC CONCEPTS AND  EXAMPLES OF ITERATIVE METHODS IN CHAPTERS REFCHAPITER1 THROUGH  REFCHAPEMENDDESCRIPTIONTO THESE MIGHT BE ADDED THE TOPICS OF MODERN ALGEBRA NUMBER THEORYCOMPLEX ANALYSIS INTERPOLATION AND APPROXIMATION THEORY AND OTHERTOPICS TOO NUMEROUS TO FIT WITHIN THE COVERS OF A SINGLE BOOKTHESE TOPICS COVER A VERY LARGE TERRITORY  IN EACH OF THESE TOPICAREAS NUMEROUS VOLUMES HAVE BEEN WRITTEN  OUR INTENT IS TO NOT TOPROVIDE AN EXHAUSTIVE TREATMENT IN EACH AREA BUT TO PRESENT ENOUGHINFORMATION TO PROVIDE A USEFUL SET OF TOOLS WITH BROAD APPLICATIONOUR APPROACH IS DIFFERENT FROM MANY OTHER BOOKS ON SIGNAL PROCESSINGIN THAT WE DO NOT EXHAUSTIVELY EXAMINE A PARTICULAR DISCIPLINE OFSIGNAL PROCESSING  FOR EXAMPLE SPECTRUM ESTIMATION  BRINGING INMATHEMATICAL TOOLS AS NECESSARY TO TREAT ISSUES THAT ARISE  INSTEADWE PRESENT THE MATHEMATICAL PERSPECTIVE FIRST INTRODUCING NEW SIGNALPROCESSING PROBLEMS AND ENHANCING UNDERSTANDING OF ALREADYINTRODUCEDPROBLEMS AS THE MATERIAL PERMITS  BY THIS MEANS PARALLELS MAY BEDRAWN BETWEEN AREAS THAT SHARE MATHEMATICAL TOOLS BUT THAT ARE NOTCOMMONLY PRESENTED TOGETHERSECTIONMATHEMATICAL MODELSTHROUGHOUT MOST OF THE REMAINDER OF THIS CHAPTER WE PRESENT EXAMPLESSEVERAL DIFFERENT MODELS THAT ARE COMMONLY USED IN SIGNAL PROCESSINGTHE MODELS ARE ROUGHLY CATEGORIZED AS FOLLOWSBEGINENUMERATEITEM LINEAR SIGNAL MODELS FOR DISCRETE AND CONTINUOUS TIME INCLUDING  TRANSFER FUNCTION AND STATE SPACE REPRESENTATIONS ALSO  APPLICATIONS OF THESE MODELS TO SIGNAL PROCESSING PROBLEMS SUCH AS  PREDICTION OR SPECTRUM ESTIMATIONITEM ADAPTIVE FILTERING MODELS AND APPLICATIONS TO PREDICTION  SYSTEM IDENTIFICATION ETCITEM THE GAUSSIAN RANDOM VARIABLE RV INCLUDING THE IMPORTANT  IDEA OF CONDITIONING UPON AN OBSERVATIONITEM HIDDEN MARKOV MODELSENDENUMERATETHESE EXAMPLES ILLUSTRATE SOME OF THE NOTATION USED THROUGHOUT THISBOOK AND PROVIDE A STARTING POINT FOR SEVERAL OF THE SIGNALPROCESSING APPLICATIONS THAT ARE EXAMINED  THUS THE MATERIAL MOSTLYSETS THE STAGE POSING QUESTIONS AND INTRODUCING ASSOCIATED WITH THEMODELS LEAVING THE QUESTIONS TO BE ANSWERED IN LATER CHAPTERS  THEMATERIAL HERE IS PRESENTED PARTLY BY WAY OF REVIEW AND PARTLY AS APARTIAL SURVEY AND MOTIVATOR OF CONCEPTS TO BE DEVELOPED THROUGHOUTTHE BOOK  SEVERAL NEW IDEAS ARE TOUCHED ON HERE THOUGH WITH THEINTENT THAT IT WILL MOTIVATE AND FORESHADOW THE TOPICS IN UPCOMINGCHAPTERSAFTER THIS INTRODUCTORY MATERIAL WE PRESENT A DISCUSSION OF PROOFSTHE CHAPTER ENDS WITH THE DEVELOPMENT OF A FAST ALGORITHM  FINALLYAN ALGORITHM  FOR SOLUTION OF A SYSTEM OF TOEPLITZ EQUATIONSTHIS ALGORITHM  MORE COMMONLY DISCUSSED IN THE ERROR CONTROLLITERATURE THAN THE SIGNAL PROCESSING LITERATURE  TIES TOGETHERSEVERAL THEMES OF THE CHAPTER LINEAR SYSTEMS NOTATION AUTOREGRESSIVEMODELS ALGORITHMS AND PROOFSSECTIONMODELS FOR LINEAR SYSTEMS AND SIGNALSLABELSECLTIMOST OF THE SYSTEMS TREATED IN SIGNAL PROCESSING ARE ASSUMED TO BELINEARINDEXLINEAR SYSTEM A CONCEPT THAT SHOULD BE FAMILIAR FROMINTRODUCTORY SIGNAL PROCESSING COURSES  WE WILL FOCUS PRINCIPALLY ONSYSTEMS THAT ARE ALSO TIME INVARIANT SUCH SYSTEMS ARE SAID TO BELINEAR TIMEINVARIANT LTI  SYSTEMS ARE DIVIDED ACCORDING TO WHETHERTHEY OPERATE IN CONTINUOUS TIME OR DISCRETE TIME  IN DISCRETE TIMETHE DATA ASSOCIATED WITH TIME T ARE INDICATED BY EITHER SQUAREBRACKETS INDEX  SQUARE BRACKETSDISCRETETIMESUCH AS XT OR BY SUBSCRIPTS SUCH AS XT WHERE TIS AN INTEGER  WE WILL ALSO EMPLOY OTHER VARIABLES AS A DISCRETETIMEINDEX SUCH AS N OR K  FOR CONTINUOUSTIME SIGNALS THE NOTATIONXT OR XT IS COMMONLY EMPLOYED WHERE T IS A REAL NUMBERINDEX CONTINUOUS TIMEWE WILL FIRSTPRESENT SOME CONCEPTS AND NOTATION FOR DISCRETE TIME SIGNALS ANDSYSTEMS THEN TRANSLATE THE NOTATION TO CONTINUOUSTIME  THE MATERIAL IN THIS SECTION IS INTENDED TO BE PRIMARILY AS A REVIEWTHIS SECTION IS FAIRLY LONG DUE TO THE IMPORTANCE OF THE MATERIAL AND THENUMBER OF INTERESTING PROBLEMS IT INTRODUCESSUBSECTIONLINEAR DISCRETETIME MODELSSUBSUBSECTIONDIFFERENCE EQUATIONSLET FT DENOTE THE SCALAR INPUT TO A DISCRETETIME LINEAR SYSTEMAND LET YT DENOTE THE SCALAR OUTPUT  IT IS COMMON TO ASSUME ANINPUTOUTPUT RELATION OF THE FORM OF THE INDEXDIFFERENCE EQUATIONDIFFERENCE EQUATIONBEGINMULTLINE YT  ABAR1 YT1  ABAR2 YT2  CDOTS  ABARP YTP    BBAR0 FT  BBAR1 FT1  CDOTS  BBARQ FTQLABELEQARMA0ENDMULTLINETHE EQUATION IS SHOWN UNDER GENERAL ASSUMPTION OF COMPLEX SIGNALS ANDTHE BAR OVER THE COEFFICIENTS DENOTES EM COMPLEX CONJUGATION  SEEBOX REFBOXCOMPLEXNOT  BY REDEFINING EACH COEFFICIENT ABARIAND BBARI IN TERMS OF ITS CONJUGATE REFEQARMA0 COULD ALSOBE WRITTEN WITHOUT THE CONJUGATES ASBEGINMULTLINE YT  A1 YT1  A2 YT2  CDOTS  AP YTP    B0 FT  B1 FT1  CDOTS  BQ FTQ NONUMBERENDMULTLINEWITH CONSISTENT AND CAREFUL USE OF THE NOTATION THE QUESTION OFWHETHER THE COEFFICIENTS ARE CONJUGATED IN THE DEFINITION OF THELINEAR MODEL IS OF NO ULTIMATE SIGNIFICANCE  THE ANSWERS OBTAINEDARE INVARIABLY THE SAME  HOWEVER THE BULK OF SIGNAL PROCESSINGLITERATURE SEEMS TO FAVOR THE CONJUGATED REPRESENTATION INREFEQARMA0  OF COURSE WHEN THE SIGNALS AND COEFFICIENTS ARESTRICTLY REAL THE CONJUGATION IS SUPERFLUOUS AND THE SYSTEM CAN ALSOBE WRITTEN IN THE FORMBEGINMULTLINE YT  A1 YT1  A2 YT2  CDOTS  AP YTP    B0 FT  B1 FT1  CDOTS  BQ FTQ NONUMBERENDMULTLINEWITHOUT THE CONJUGATES ON THE COEFFICIENTSBEGINTEXTBOX09TEXTWIDTHNOTATION FOR COMPLEX QUANTITIESLABELBOXCOMPLEXNOTINDEXCOMPLEX CONJUGATE INDEXBAROVERLINE SEECOMPLEX CONJUGATEWE USE THE ENGINEERS NOTATION JSQRT1 RATHER THAN THEMATHEMATICIANS I  HOWEVER IN SOME PLACES J WILL BE USED AS ANINDEX OF SUMMATION CONTEXT SHOULD MAKE CLEAR WHAT IS INTENDEDINDEXJJ INDEXIIA BAR OVER A QUANTITY DENOTES EM COMPLEX CONJUGATION  OTHERAUTHORS COMMONLY INDICATE COMPLEX CONJUGATION USING A SUPERSCRIPTASTERISK AS A  HOWEVER THE ABAR NOTATION IS USED IN THISBOOK TO INDICATE CONJUGATION SINCE A IS ALSO COMMONLY USED TODENOTE A PARTICULAR VALUE OF A SUCH AS A MINIMIZING VALUE OR TOINDICATE THE ADJOINT OF A LINEAR OPERATORENDTEXTBOXIN THE CASE OF A SYSTEM THAT IS NOT TIMEINVARIANT THE COEFFICIENTSMAY BE A FUNCTION OF THE TIME INDEX T  WE WILL ASSUME FOR THE MOSTPART CONSTANT COEFFICIENTS  THE RELATION REFEQARMA0 CAN BEWRITTEN ASBEGINEQUATION SUMK0P ABARK YTK  SUMK0Q BBARK FTKLABELEQARMAENDEQUATIONWITH A0  1  IN REFEQARMA WHEN P0BEGINEQUATIONYT  SUMK0Q BBARK FTKLABELEQFIR1ENDEQUATIONTHE SIGNAL YT IS CALLED IN THE STATISTICAL LITERATURE A EM  MOVING AVERAGE MA SIGNAL INDEXMOVING AVERAGE SINCE IT ISFORMED BY SIMPLY ADDING UP SCALED VERSIONS OF THE INPUT SIGNAL OVERA WINDOW OF Q1 VALUES  THE NUMBER Q IS THE EM ORDER OF THE MASIGNAL  THE SIGNAL IS DENOTED EITHER AS MA OR MAQINDEXMASEEMOVING AVERAGE WE CAN ALSO WRITE REFEQFIR1 USINGA CONVENIENT VECTOR NOTATION  LET FBFT  BEGINBMATRIX FT  FT1  VDOTS  FTQENDBMATRIX QQUAD TEXTAND QQUADBBF  BEGINBMATRIX B0  B1  VDOTS  BQ ENDBMATRIXTHEN  YT  BBFH FBFT  OVERLINEFBFTTBBFTHE VECTOR NOTATION USED IN THIS EXAMPLE IS SUMMARIZED IN BOXREFBOXVECTORNOTINDEXBOLD FONTSEEFONTSIN REFEQARMA WHEN Q0 SO THAT YT  BBAR0 UN SUMK1P ABARK YTKTHE SIGNAL Y IS SAID TO AN EM AUTOREGRESSIVE AR SIGNAL OF ORDER PINDEXARSEEAUTOREGRESSIVEINDEXAUTOREGRESSIVE  EM AUTO BECAUSE IT EXPRESSES THE SIGNAL IN TERMS OF ITSELF  EM REGRESSIVE IN THE SENSE THAT A FUNCTIONAL RELATIONSHIP EXISTS  BETWEEN TWO   OR MORE VARIABLES  AN AUTOREGRESSIVE MODEL IS DENOTED AS AR OR  ARP  WRITING YBFT  BEGINBMATRIX YT1  YT2  VDOTS  YTPENDBMATRIX QQUAD TEXTAND QQUADABF  BEGINBMATRIX A1  A2  VDOTS  AP ENDBMATRIXWE CAN WRITE THE AR SIGNAL AS YT  BBAR0 UT  ABFH YBFTBEGINTEXTBOX09TEXTWIDTHNOTATION FOR VECTORSLABELBOXVECTORNOTBEGINENUMERATEITEM VECTORS IN A FINITEDIMENSIONAL VECTOR SPACE ARE TYPICALLY  DENOTED IN BOLD FONT SUCH AS BBF  INDEXFONTSBOLDITEM ALL VECTORS IN THIS BOOK ARE ASSUMED TO BE COLUMN VECTORS  IN  SOME CASES A VECTOR WILL BE TYPESET IN HORIZONTAL FORMAT WITH T  TRANSPOSE TO INDICATE THAT IT SHOULD BE TRANSPOSED  THUS WE COULD  HAVE EQUIVALENTLY WRITTEN BBF  B0 B1 LDOTS BQTQQUAD TEXTORQQUADBBFT  B0B1 LDOTS BQINDEXTTINDEXHHINDEXTRANSPOSETITEM IN GENERAL THE ITH COMPONENT OF A VECTOR BBF WILL BE  DESIGNATED AS BI  WHETHER THE INDEX I STARTS WITH 0 OR 1 OR  SOME OTHER VALUE DEPENDS ON THE NEEDS OF THE PARTICULAR PROBLEMITEM THE NOTATION BBFH DENOTES THE EM HERMITIAN TRANSPOSE IN  WHICH BBF IS TRANSPOSED AND ITS ELEMENTS ARE CONJUGATEDINDEXTRANSPOSEH HERMITIAN BBFH  BBAR0 BBAR1 LDOTS BBARQENDENUMERATETHESE RULES NOTWITHSTANDING FOR NOTATIONAL CONVENIENCE WE WILLSOMETIMES DENOTE THE VECTOR WITH N ELEMENTS AS AN NTUPLE SO THAT XBF  BEGINBMATRIX X1  X2  LDOTS  XN ENDBMATRIXTQQUADTEXTANDQQUAD XBF  X1X2LDOTSXNARE OCCASIONALLY USED SYNONYMOUSLY  THIS NTUPLE NOTATION IS USEDPARTICULARLY WHEN XBF IS REGARDED AS A POINT IN RBBNFURTHERMORE SINCE WE WILL GENERALIZE THE CONCEPT OF VECTORS TOINCLUDE FUNCTIONS THE MATH ITALIC NOTATION X WILL BE USED IN THEMOST GENERAL CASE TO REPRESENT VECTORS EITHER IN RBBN OR ASFUNCTIONSINDEXFONTSMATH ITALICINDEXTTSEETRANSPOSEINDEXHHSEETRANSPOSEBOXINDENT MATRICES ARE REPRESENTED WITH CAPITAL LETTERS AS IN A OR X  THEMATRIX I IS AN IDENTITY MATRIXINDEXIIINDEXVECTOR NOTATIONINDEXMATRIX NOTATIONTHE NOTATION ZEROBF IS USED TO INDICATE A VECTOR OR MATRIX OFZEROS WITH THE SIZE DETERMINED BY CONTEXT INDEX0ZEROBFSIMILARLY THE NOTATION ONEBF IS USED TO INDICATE A VECTOR ORMATRIX OF ONES WITH THE SIZE DETERMINED BY CONTEXT INDEX1ONEBFENDTEXTBOXTHE GENERAL FORM IN REFEQARMA COMBINING BOTH THE AUTOREGRESSIVEAND THE MOVING AVERAGE COMPONENTS IS CALLED AN EM AUTOREGRESSIVE  MOVINGAVERAGE OR ARMA OR ARMAPQ  WHERE ALL THE SIGNALSARE DETERMINISTIC THE TERM DARMA DETERMINISTIC ARMA IS SOMETIMESEMPLOYED INDEXARMA INDEXAUTOREGRESSIVE MOVING AVERAGESUBSUBSECTIONSYSTEM FUNCTION AND IMPULSE RESPONSEIN THE INTEREST OF GETTING A SYSTEM FUNCTION THAT DOES NOT DEPEND UPONINITIAL CONDITIONS WE ASSUME THAT THE INITIAL CONDITIONS ARE ZERO ANDTAKE THE ZTRANSFORM TO OBTAIN YZ SUMK0P ABARK ZK  FZ SUMK0Q BBARK ZKWHICH WE WRITE AS YZ AZ  FZ BZWE WILL OCCASIONALLY WRITE THE TRANSFORM RELATIONSHIP AS YT LEFTRIGHTARROW YZWHERE THE PARTICULAR TRANSFORM INTENDED IS DETERMINED BYCONTEXT INDEXLEFTRIGHTARROW  WE WILL ALSO DENOTEZTRANSFORMS BY YZ  ZCYT INDEXZZCTHE EM SYSTEM FUNCTION INDEXSYSTEM FUNCTIONSEETRANSFER FUNCTION ISBEGINEQUATION HZ  FRACYZFZ  FRACSUMK0Q BBARK  ZKSUMK0P ABARK ZK   FRACSUMK0Q BBARK  ZK1 SUMK1P ABARK ZK  FRACBZAZLABELEQHZ1ENDEQUATIONTHIS IS ALSO CALLED USUALLY INTERCHANGEABLY THE EM TRANSFER  FUNCTION INDEXTRANSFER FUNCTION OF THE SYSTEM  WE WRITEBEGINEQUATION YZ  HZ FZLABELEQYHZ1ENDEQUATIONAND REPRESENT THIS AS SHOWN IN FIGURE REFFIGSYST1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRSYST1LATEXINPUTPICTUREDIRSYST1    CAPTIONINPUTOUTPUT RELATION FOR A TRANSFER FUNCTION    LABELFIGSYST1  ENDCENTERENDFIGUREIF THE SYSTEM IS AR THEN HZ  FRAC11 SUMK1P ABARK ZK  FRAC1AZAND HZ IS SAID TO BE AN EM ALLPOLESYSTEM INDEXALLPOLESEEAUTOREGRESSIVEIF THE SYSTEM ISMA THEN HZ  SUMK0Q BBARK  ZK  BZWHICH IS CALLED AN EM ALLZERO SYSTEM  INDEXALLZEROSEEMOVING    AVERAGE THE CORRESPONDING INDEXDIFFERENCE EQUATION DIFFERENCE    EQUATION REFEQFIR1 HAS ONLY A FINITE NUMBER OF NONZERO OUTPUTS WHEN THE INPUT IS A DELTAFUNCTION FT  DELTAT WHERE  INDEXDELTA FUNCTION  DELTAT  BEGINCASES  1  T  0   0  T NEQ 0ENDCASESWE WILL ALSO WRITE THE DELTA FUNCTION AS DELTAT  OCCASIONALLY THEFUNCTION DELTATTAU WILL BE WRITTEN AS DELTATTAUA SYSTEM THAT HAS ONLY A FINITE NUMBER OF NONZERO OUTPUTS INRESPONSE TO A DELTA FUNCTION IS REFERRED TO AS A INDEXFINITE IMPULSE RESPONSE FINITE IMPULSE  RESPONSE FIR SYSTEM  INDEXFIRSEEFINITE IMPULSE RESPONSE ASYSTEM WHICH IS NOT FIR IS INDEXINFINITE IMPULSE RESPONSE INFINITEIMPULSE RESPONSE IIRWE CAN VIEW SIGNAL YZ AS THE OUTPUT OF A SYSTEM WITH SYSTEM FUNCTIONHZ DRIVEN BY AN INPUT FZ  TAKING THE INVERSE ZTRANSFORM OFREFEQYHZ1 AND RECALLING THE CONVOLUTION PROPERTY INDEXCONVOLUTIONMULTIPLICATION IN THE TRANSFORM DOMAIN CORRESPONDS TO CONVOLUTION INTHE TIME DOMAIN WE OBTAIN YT  SUMKINFTYINFTY UK HTKWHERE HT THE IMPULSE RESPONSEINDEXIMPULSE RESPONSE IS THE INVERSE TRANSFORM OFHZ    TO COMPUTE THE INVERSE TRANSFORM OF HZ WE FIRST FACTOR HZINTO MONOMIAL FACTORS USING THE ROOTS OF THE NUMERATOR AND DENOMINATORPOLYNOMIALS HZ  FRACBBAR0 PRODK1Q 1ZI Z1PRODK1P 1PI Z1  FRACBZAZWHERE THE ZI ARE THE NONZERO ROOTS OF BZ CALLED THE EM  ZEROS OF THE SYSTEM FUNCTION AND THE PI ARE THENONZERO ROOTS OF AZ CALLED THE EM POLES INDEXPOLE OF THESYSTEM FUNCTION  IN THIS FORM WE OBSERVE THAT IF A POLE IS EQUAL TOA ZERO THE FACTORS CAN BE CANCELED OUT OF BOTH THE NUMERATOR ANDDENOMINATOR TO OBTAIN AN EQUIVALENT TRANSFER FUNCTION  A WORD OFCAUTION EVEN THOUGH TERMS MAY CANCEL FROM THE NUMERATOR ANDDENOMINATOR AS SEEN FROM THE TRANSFER FUNCTION THE PHYSICALCOMPONENTS THAT THESE TERMS MODEL MAY STILL EXIST AND COULD INTRODUCEDIFFICULTY  A SYSTEM WITH THE SMALLEST DEGREE NUMERATOR ANDDENOMINATOR IS SAID TO BE A EM MINIMAL SYSTEM INDEXMINIMAL  SYSTEMBEGINEXAMPLETHE SYSTEM FUNCTION HZ  FRAC1 7Z1  12 Z215Z1  06Z2CAN BE FACTORED AS HZ  FRAC13Z114Z112Z113Z1  FRAC14Z112Z1THUS THE HZ IS NOT A MINIMAL REALIZATIONENDEXAMPLESUBSUBSECTIONPARTIAL FRACTION EXPANSION PFEASSUMING FOR THE MOMENT THAT THE POLES ARE UNIQUE NO REPEATED POLESAND THAT QP THEN BY PARTIAL FRACTION EXPANSION PFEINDEXPARTIAL FRACTION EXPANSION PFE THE SYSTEM FUNCTION CAN BEEXPRESSED ASBEGINEQUATIONHZ  SUMK1P FRACNK1PK Z1LABELEQHZ2ENDEQUATIONWHERE NK  HZ1PK Z1BIGRZPKTAKING THE CAUSAL INVERSE ZTRANSFORM OF REFEQHZ2 WE OBTAIN HT  SUMK1P NK PKTQQUAD T GEQ 0THE FUNCTIONS PKN ARE THE NATURAL MODES INDEXMODE OF THE SYSTEMHZ  CLEARLY FOR THE CAUSAL MODES TO BE BOUNDED IN TIME WE MUSTHAVE PK LEQ 1 INDEXSTABILITYIN GENERAL THE OUTPUT OF A LINEAR TIMEINVARIANT SYSTEM IS THE SUM OFTHE NATURAL MODES OF THE SYSTEM PLUS THE INPUT MODES OF THE SYSTEMBEGINEXAMPLE  LET HZ  FRAC13Z1 1 11Z1  3Z2  FRAC13Z115Z116Z1THEN A PARTIAL FRACTION EXPANSION IS HZ  FRAC215Z1  FRAC316Z1THE IMPULSE RESPONSE IS HT  25T  36TUTWHERE UT IS THE UNITSTEP FUNCTION INDEXUNITSTEP FUNCTION UT  BEGINCASES  1  T GEQ 1   0  T  0ENDCASESENDEXAMPLETO COMPUTE THE PFE WHEN Q GEQ P THE RATIO OF POLYNOMIALS IS FIRSTDIVIDED OUT  WHEN THERE ARE REPEATED POLES INDEXREPEATED POLESSOMEWHAT MORE CARE IS REQUIRED  FOR EXAMPLE A ROOT REPEATED RTIMES AS IN HZ  FRACBZ1P Z1RGIVES RISE TO THE PARTIAL FRACTION EXPANSIONBEGINEQUATION HZ  FRACK01P Z1R  FRACK11P Z1R1   CDOTS  FRACKR11PZ1LABELEQHRRENDEQUATIONWHEREFOOTNOTETHE SYMBOL J HERE DOES NOT REPRESENT SQRT1  IN  INSTANCES WHERE CONFUSION IS UNLIKELY WE MAY USE J AS AN INDEX  VALUEBEGINEQUATIONKJ  FRAC1PJ J1J FRACDJDZ1J 1PZ1RHZLABELEQPFEZTENDEQUATIONTHE INVERSE ZTRANSFORM CORRESPONDING TO REFEQHRR IS OF THE FORM HT  C0PT  C1 T PT  CDOTS  CR1 TR PT UTWHERE THE COEFFICIENTS CI ARE LINEARLY RELATED TO THE PFECOEFFICIENTS KIUSING COMPUTER SOFTWARE TO COMPUTE PARTIAL FRACTION EXPANSIONS SUCHAS THE TT RESIDUE OR TT RESIDUEZ COMMAND IN SC MATLAB ISRECOMMENDEDBEGINEXAMPLELET HZ  FRAC3  24 Z1  6 Z217Z1  1Z2WE DESIRE TO FIND THE IMPULSE RESPONSE HT  SINCE THE DEGREE OFTHE NUMERATOR IS THE SAME AS THE DEGREE OF THE DENOMINATOR WE DIVIDETHEN FIND THE PARTIAL FRACTION EXPANSION BEGINALIGNEDHZ  60  FRAC444Z1  5712Z115Z1  60  FRAC11012Z1  FRAC5315Z1ENDALIGNEDTHEN HT  60 DELTAT  1102T  535T QQUAD T GEQ 0ENDEXAMPLEBEGINEXERCISESITEM SHOW THAT REFEQPFEZT FOR THE PARTIAL FRACTION EXPANSION OF  A ZTRANSFORM WITH REPEATED ROOTS IS CORRECTITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF THE FOLLOWINGBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2TEXTC  HZ  FRAC2  3Z113 Z12 TEXTD  HZ  FRAC56Z113 Z1214Z1ENDARRAYCHECK YOUR RESULTS USING TT RESIDUEZ IN SC MATLABITEM INVERSES OF HIGHERORDER MODES  BEGINENUMERATE  ITEM PROVE THE FOLLOWING PROPERTY FOR Z TRANSFORMS  IF XT LEFTRIGHTARROW XZTHEN TXT LEFTRIGHTARROW Z FRACD XZDZITEM USING THE FACT THAT PT UT LEFTRIGHTARROW 11PZ1  SHOW THAT T PT UT LEFTRIGHTARROW FRACPZ11PZ12ITEM DETERMINE THE Z TRANSFORM OF T2 PT UTITEM BY EXTRAPOLATION DETERMINE THE ORDER OF THE POLE OF A MODE OF  THE FORM TK PT UT  ENDENUMERATEENDEXERCISESSUBSECTIONSTOCHASTIC MA AND AR MODELSLABELSECARPROCESSIN STOCHASTIC INDEXSTOCHASTIC PROCESSES MA AND AR MODELS THE INPUTFT IS ASSUMED TO TO BE A WHITE DISCRETETIME RANDOM PROCESS THATIS USUALLY ZERO MEAN  THE READER IS ENCOURAGED TO REVIEW THECONCEPTS OF RANDOM PROCESSES SUMMARIZED IN APPENDIX REFAPPDXRPTHE INPUT COEFFICIENT B0 IS SET TO 1 WITH THE INPUT POWERDETERMINED BY THE VARIANCE OF THE SIGNAL  THUS EFT  0 QQUAD TEXTFOR ALL  TAND EFT FBARS  BEGINCASES  SIGMAF2  T  S 0  TEXTOTHERWISEENDCASESSUBSUBSECTIONAUTOCORRELATION FUNCTIONSIGNAL PROCESSING OFTEN INVOLVES COMPARING TWO SIGNALS ONE MEANS OFCOMPARISON IS BY MEANS OF CORRELATION  WHEN A SIGNAL IS COMPARED WITHITSELF THE INDEXCORRELATION CORRELATION IS CALLEDAUTOCORRELATIONINDEXAUTOCORRELATION  FOR STOCHASTIC SIGNALS WEDEFINE THE AUTOCORRELATION OF A ZEROMEAN WIDESENSE STATIONARYSIGNAL YT ASBEGINEQUATION RYYLK  EYTK YBARTLLABELEQAUTOCORRDEFENDEQUATIONOR EQUIVALENTLY RYYK  EYT YBARTK  THEAUTOCORRELATION FUNCTION HAS THE PROPERTY THATBEGINEQUATION RYYK  RBARYYKLABELEQRHERMENDEQUATIONFOR REAL RANDOM PROCESSES RYYK  RYYK AN EVEN FUNCTIONOF K INDEXEVEN  FUNCTIONFOR THE MA PROCESS YT  FT  BBAR1 FT1  CDOTS BBARQ FTQIT IS STRAIGHTFORWARD TO SHOW THAT THE AUTOCORRELATION FUNCTION ISBEGINEQUATION RYYK  SIGMAF2 SUML BLK BBARLLABELEQMAAUTOCORRENDEQUATIONWHERE THE SUM IS OVER ALL VALUES L SUCH THAT BL OR BLK ARENOT ZERO AND B0  1  FOR THE AR MODELBEGINEQUATION YT  ABAR1 YT1  CDOTS  ABARP YTP  FTLABELEQAR2ENDEQUATIONMULTIPLY BOTH SIDES BY YBARTL AND TAKE EXPECTATIONS TO OBTAINBEGINEQUATION  LABELEQYW1  ELEFTSUMK0P ABARK YTKYBARTLRIGHT  EFT  YBARTLENDEQUATIONWE RECOGNIZE THAT EYTK YBARTL  RYYLKAND THAT THE RIGHTHAND SIDE EFTYBARTL  0FOR L0 SINCE FT IS A WHITENOISE PROCESS  THEN USING THE FACTTHAT A01 WE CAN WRITEBEGINEQUATION RYYL  ABAR1 RYYL1  ABAR2 RYYL2  CDOTS ABARP RYYLPQQUADTEXT FOR  L  0LABELEQAR3ENDEQUATIONTHIS DIFFERENCE EQUATION FOR THE AUTOCORRELATION IS SIMILAR TO THE EQUATION FORTHE ORIGINAL DIFFERENCE EQUATION IN REFEQAR2  STACKINGREFEQAR3 FOR L12LDOTSP WE OBTAIN BEGINEQUATION  LABELEQYW2  BEGINBMATRIX RYY0 RYY1  CDOTS  RYYP1 RYY1  RYY0  CDOTS  RYYP2 VDOTS RYYP1  RYYP2  CDOTS  RYY0  ENDBMATRIXBEGINBMATRIX ABAR1  ABAR2  VDOTS  ABARPENDBMATRIX BEGINBMATRIX RYY1  RYY2  VDOTS  RYYP ENDBMATRIXENDEQUATIONCONJUGATING BOTH SIDES USING REFEQRHERM WE OBTAINBEGINEQUATION  LABELEQYW3  BEGINBMATRIX RYY0 RYY1  CDOTS  RYYP1 RBARYY1  RYY0  CDOTS  RYYP2 VDOTS RBARYYP1  RBARYYP2  CDOTS  RYY0  ENDBMATRIXBEGINBMATRIX A1  A2  VDOTS  APENDBMATRIX BEGINBMATRIX RBARYY1  RBARYY2  VDOTS   RBARYYP ENDBMATRIX ENDEQUATIONTHESE EQUATIONS ARE KNOWN AS THE EM YULEWALKER EQUATIONSINDEXYULEWALKER EQUATIONS  WE COMMONLY WRITE REFEQYW3 AS R WBF  RBFWHERE WBF  BEGINBMATRIX A1  A2 CDOTS  APENDBMATRIXTQQUAD RBF  BEGINBMATRIXRBARYY1  RBARYY2  CDOTS   RBARYYP ENDBMATRIXTHE MATRIX R IS SAID TO BE THE EM AUTOCORRELATION MATRIX OF YTHROUGHOUT THE BOOK WE WILL HAVE CONSIDERABLE TO SAY ABOUT THEPROPERTIES OF R AND ALGORITHMS THAT OPERATE ON IT  FOR NOW WE MAKETHE FOLLOWING OBSERVATIONSBEGINENUMERATEITEM R IS EM HERMITIAN SYMMETRIC INDEXHERMITIAN    SYMMETRICSEESYMMETRIC INDEXSYMMETRIC WHICH MEANS THAT R  RHWE WILL SEE THAT THIS MEANS THAT THE EIGENVALUES OF R ARE REAL ANDTHE EIGENVECTORS CORRESPONDING TO DISTINCT EIGENVALUES ARE ORTHOGONALIF R IS REAL THEN R IS EM SYMMETRIC RT  RITEM R IS A EM TOEPLITZ MATRIX INDEXTOEPLITZ MATRIX WHICH  MEANS THAT R IS CONSTANT ALONG THE DIAGONALS  IF RIJ DENOTES  THE IJTH  ELEMENT OF R THEN RJJ  RIJTHE ELEMENTS OF R DEPENDS ONLY ON THE DIFFERENCE BETWEEN THE INDEXVALUES  WE SHALL SEE THAT THE TOEPLITZ STRUCTURE OF R LEADS TOEFFICIENT ALGORITHMS FOR SOLVING EQUATIONS SIMILAR TO THE YULEWALKEREQUATIONSENDENUMERATEBEGINEXERCISESITEM SHOW THAT THE AUTOCORRELATION FUNCTION DEFINED IN  REFEQAUTOCORRDEF HAS THE PROPERTY THAT  RYYK  RBARYYKITEM SHOW THAT REFEQMAAUTOCORR IS CORRECTITEM FOR THE MA PROCESS YT  FT  2FT1  3FT2WHERE FT IS ZEROMEAN WHITE RANDOM PROCESS WITH SIGMAF2  1DETERMINE THE MATSIZE44 AUTOCORRELATION MATRIX RITEM FOR THE FIRSTORDER REAL AR PROCESS YT1  A1 YT   FT1WITH A11 WITH EFT  0 SHOW THATBEGINEQUATIONSIGMAY2  EY2T  FRACSIGMAF21A12LABELEQFIRSTARVARENDEQUATIONITEM FOR AN AR PROCESS REFEQAR2 DRIVEN BY A WHITENOISE  SEQUENCE FT WITH VARIANCE SIGMAF2 SHOW THATBEGINEQUATIONSIGMAF2  SUMI0P AI RYYI HAYKIN P 120LABELEQARINPUTVARENDEQUATIONITEM LET YT  7YT1  12 Y2T  FT WHERE FT IS A  ZEROMEAN WHITENOISE RANDOM PROCESS WITH SIGMAF2  2    BEGINENUMERATE  ITEM WRITE  THE YULEWALKER EQUATIONS FOR Y  ITEM DETERMINE RYY1 AND RYY2  ITEM FIND SIGMAY2  ENDENUMERATEITEM CONSIDER THE SECONDORDER REAL AR PROCESSBEGINEQUATION YT2  A1 YT1  A2 YT  FT2LABELEQYULEWALKER21ENDEQUATIONWHERE FT IS A ZEROMEAN WHITENOISE SEQUENCE  THE DIFFERENCEEQUATION IN REFEQAR3 HAS A CHARACTERISTIC EQUATION WITH ROOTS P1 P2  FRAC12A1 PM SQRTA12  4A2BEGINENUMERATEITEM USING THE YULEWALKER EQUATIONS SHOW THAT IF THE  AUTOCORRELATION VALUES  RYYLK  EYTKYBARTLARE KNOWN THEN THE MODEL PARAMETERS MAY  BE DETERMINED FROMBEGINEQUATIONBEGINSPLITA1  FRACR1R0  R2 R20   R21 A2  FRACR0R2  R21 R20   R21ENDSPLITLABELEQYW5ENDEQUATIONITEM ON THE OTHER HAND IF SIGMAY2  R0 AND A1 AND  A2 ARE KNOWN SHOW THAT THE AUTOCORRELATION VALUES CAN BE  EXPRESSED AS  BEGINEQUATION    LABELEQYW6BEGINSPLIT    RYY1  FRACA11A2 SIGMAY2RYY2  SIGMAY2LEFT FRACA121A2  A2RIGHTENDSPLITENDEQUATIONITEM USING REFEQARINPUTVAR AND THE RESULTS OF THIS PROBLEM  SHOW THAT  BEGINEQUATION    LABELEQYW7    RYY0  SIGMAY2  LEFTFRAC1A21A2RIGHT    FRACSIGMAF21A22  A12  ENDEQUATIONITEM USING RYY0  SIGMAY2 AND RYY1  A1  SIGMAY21A2 AS INITIAL CONDITIONS FIND AN EXPLICIT SOLUTION  TO THE YULEWALKER DIFFERENCE EQUATION RYYK  A1 RYYK1  A2 RYYK2  0IN TERMS OF P1 P2 AND SIGMAY2 HAYKIN P 121ENDENUMERATEITEM FOR THE SECONDORDER DIFFERENCE EQUATION YT2  7 YT1 12 YT  FT2WHERE FT IS A ZEROMEAN WHITE SEQUENCE WITH SIGMAF2  1DETERMINE SIGMAY2  RYY0 RYY1 AND RYY2ITEM A RANDOM PROCESS YT HAVING ZEROMEAN AND MATSIZEMM  AUTOCORRELATION MATRIX R IS APPLIED TO AN FIR FILTER WITH IMPULSE  RESPONSE VECTOR HBF  H0H1H2LDOTSHM1  DETERMINE  THE AVERAGE POWER OF THE FILTER OUTPUTENDEXERCISESSUBSECTIONREALIZATIONSA BLOCK DIAGRAM OR EM REALIZATION OF REFEQARMA CAN BEEASILY DERIVED  THE REALIZATION PRESENTED HERE ISKNOWN IN THE CONTROL LITERATURE AS THE EM CONTROLLER CANONICAL  FORM  INDEXCONTROLLER CANONICAL FORM WRITE THE SYSTEM FUNCTION ASBEGINEQUATION HZ  FRACYZWZ FRACWZFZ  LEFTSUMK0Q BBARK  ZK RIGHT LEFTFRAC11    SUMK1P ABARK ZKRIGHT  H1Z H2ZLABELEQHZ2AENDEQUATIONWHERE THE SIGNAL WZ HAS BEEN ARTIFICIALLY INTRODUCED  FROM THETRANSFER FUNCTION H2Z WE GET THE RELATIONSHIPBEGINEQUATIONWZ1SUMK1P ABARK ZK  FZLABELEQTRANSFER1ENDEQUATIONCORRESPONDING TO THE DIFFERENCE EQUATIONWT  ABAR1 WT1  CDOTS  ABARP WTP  FTOR WT  FT  ABAR1 WT1  ABAR2 WT2  LDOTS  ABARP WTPA BLOCK DIAGRAM OF A  REALIZATION OF REFEQTRANSFER1 IS SHOWN IN FIGUREREFFIGTRANSFER1BEGINFIGURETBP  BEGINCENTER INPUTPICTUREDIRTRANSFER1LATEX INPUTPICTUREDIRTRANSFER1  CAPTIONREALIZATION OF THE AR PART OF A TRANSFER FUNCTION  LABELFIGTRANSFER1ENDCENTERENDFIGUREFROM H1Z IN REFEQHZ2A WE HAVE YZ  WZBZWITH THE CORRESPONDING DIFFERENCE EQUATION YT  BBAR0 WT  BBAR1 WT1  CDOTS  BBARQ WTQTHIS REALIZATION DRAWN ASSUMING THAT PQ IS SHOWN IN FIGUREREFFIGTRANSFER2 BEGINFIGUREHTBP  BEGINCENTERINPUTPICTUREDIRTRANSFER2LATEXINPUTPICTUREDIRTRANSFER2  CAPTIONCONTROLLER CANONICAL REALIZATION OF A TRANSFER FUNCTION  LABELFIGTRANSFER2ENDCENTERENDFIGUREWE EXPLORE OTHER POSSIBLE REALIZATIONS IN THE EXERCISESSUBSUBSECTIONSTATESPACE FORMINDEXSTATESPACE FORM CONSIDER THE BLOCK DIAGRAM IN FIGUREREFFIGTRANSFER3 IN WHICH THE OUTPUTS OF THE DELAY BLOCKS ARELABELED X1 X2 LDOTS XP FROM RIGHT TO LEFTBEGINFIGUREHTBP  BEGINCENTERINPUTPICTUREDIRTRANSFER3LATEXINPUTPICTUREDIRTRANSFER3  CAPTIONREALIZATION OF A TRANSFER FUNCTION WITH STATE VARIABLE LABELS  LABELFIGTRANSFER3ENDCENTERENDFIGUREFROM THIS BLOCK DIAGRAM WE OBTAIN THE FOLLOWING EQUATIONSBEGINEQUATIONBEGINSPLITX1T1  X2T X2T1  X3T  EQSKIP VDOTS XP1T1  XPT XPT1  FT ABAR1 XPT ABAR2 XP1T  CDOTS ABARP1 X2T  ABARP X1T SMALLSKIP YT  BBARP X1T  BBARP1 X2T  CDOTS  BBAR2XP1T  BBAR1 XPT    QQUAD BBAR0FT  ABAR1 XPT  ABAR2 XP1T  CDOTS ABARP X1TENDSPLITLABELEQSTATE1ENDEQUATIONOBSERVE THAT THE DIRECT CONNECTION FROM INPUT F TO OUTPUT Y IS VIAB0  THE VARIABLES X1 X2 LDOTS XP ARE THE BF STATE  VARIABLES  LET XBFT BE THE BF STATE VECTOR XBFT  BEGINBMATRIX X1T  X2T  VDOTS  XPTENDBMATRIX WE ALSO INTRODUCE THE VECTORSBBF  UNDERBRACE00LDOTS 01P TEXT ELEMENTST CBF  BEGINBMATRIX BBARP  BBAR0 ABARP  BBARP1   BBAR0 ABARP1   VDOTS  BBAR1  BBAR0ABAR1  ENDBMATRIXQQUAD TEXTANDQQUADD  BBAR0 AND THE MATRIXBEGINEQUATION A  BEGINBMATRIX0100 CDOTS  00 0010 CDOTS  00 VDOTS 0000 CDOTS  01 ABARP  ABARP1  ABARP2  ABARP3  CDOTS ABAR2  ABAR1  ENDBMATRIXLABELEQASTATEMATENDEQUATIONIF B00 THEN CBF IS CBFT  BBARPBBARP1LDOTSBBAR1WHICH EXPLICITLY DISPLAYS THE NUMERATOR COEFFICIENTS OF HZTHE EQUATIONS IN REFEQSTATE1 CAN BE WRITTEN USING THESEDEFINITIONS ASBEGINEQUATIONBEGINSPLITXBFT1  A XBFT  BBF FT YT  CBFT XBFT  D FTENDSPLITLABELEQSTATE2ENDEQUATIONAN EQUATION OF THE FORM REFEQSTATE2 IS IN EM STATESPACEFORM  THE SYSTEM IS DENOTED AS ABBFCBFTD OR WHEN D0 ASABBFCBFT  ALTHOUGH THE TRANSFORMATION FROM THE TRANSFERFUNCTION TO REFEQSTATE2 WAS MADE BY A PARTICULAR STATEASSIGNMENT THE REFEQSTATE2 IS OF GENERAL APPLICABILITY AND THEMATRICES DOES NOT NECESSARILY HAVE THE STRUCTURE OFREFEQASTATEMAT WHEN THE STATESPACE SYSTEM IN REFEQSTATE2 DOES HAVE THE AMATRIX OF THE FORM REFEQASTATEMAT THE STATESPACE SYSTEM ISSAID TO BE IN EM CONTROLLER FORM INDEXCONTROLLER FORM  THEFORM OF THE MATRIX A WITH ONES ABOVE THE DIAGONAL AND COEFFICIENTS ONTHE LAST ROW IS CALLED A FIRST EM COMPANION MATRIXCOMPANION MATRICES AREDISCUSSED IN SECTION REFSECCOMPANMAT INDEXCOMPANION MATRIXSUBSUBSECTIONSYSTEM TRANSFORMATIONS SIMILAR MATRICESTHE STATEVARIABLE REPRESENTATION IS NOT UNIQUE  IN FACT AN INFINITENUMBER OF POSSIBLE REALIZATIONS EXIST WHICH ARE MATHEMATICALLYEQUIVALENT ALTHOUGH NOT NECESSARILY IDENTICAL IN PHYSICAL OPERATIONWE CAN CREATE A NEW STATEVARIABLE REPRESENTATION BY LETTING XBF TZBF FOR ANY INVERTIBLE MATSIZEPP MATRIX T  THENREFEQSTATE2 BECOMES BEGINALIGNEDTZBFT1  ATZBFT  BBF FT YT  CBFT TZBF  D FTENDALIGNEDWHICH CAN BE WRITTEN ASBEGINEQUATION  LABELEQSTATE3  BEGINSPLITZBFT1  ABAR ZBFT  BBFBAR FT YT  CBFBART ZBFT  DBAR FTENDSPLITENDEQUATIONWHERE ABAR  T1 A T QQUAD BBFBAR  T1BBFQQUAD CBFBAR  TTCBF QQUAD DBAR  DTHE BAR DOES NOT INDICATE CONJUGATION IN THIS INSTANCE  MATRICESA AND ABAR THAT ARE RELATED AS ABAR  T1 A T ARE SAID TOBE EM SIMILAR  INDEXSIMILAR MATRIX IT IS STRAIGHTFORWARD TOSHOW THAT THE SYSTEM ABARBBFBARCBFBARTDBAR HAS THE SAMEINPUTOUTPUT RELATIONSHIPS DYNAMICS AND TRANSFER FUNCTION AS DOESTHE SYSTEM ABBFCBFTD  WHICH MEANS AS WE SHALL SEE THATA AND ABAR HAVE THE SAME EIGENVALUESSUBSUBSECTIONTIMEVARYING STATESPACE MODELWHEN THE SYSTEM IS TIMEVARYING INDEXTIMEVARYING SYSTEM THESTATESPACE REPRESENTATION ISBEGINEQUATIONBEGINSPLITXBFT1  AT XBFT  BBFT FT YT  CBFTT XBFT  DT FTENDSPLITLABELEQSTATE4ENDEQUATIONIN WHICH THE EXPLICIT DEPENDENCE OF ATBBFTCBFTTDT ONTHE TIME INDEX T IS SHOWNSUBSUBSECTIONTRANSFER FUNCTION FROM THE STATESPACE MODELTHE TIMEINVARIANT STATESPACE FORM CAN BE REPRESENTED USING ASYSTEM FUNCTION  WE CAN TAKE THE ZTRANSFORM OF REFEQSTATE2THE ZTRANSFORM OF A VECTOR IS SIMPLY THE TRANSFORM OF EACHCOMPONENT  WE OBTAIN THE EQUATIONSBEGINALIGNZ XBFZ  A XBFZ  BBF FZ LABELEQSS1 YZ  CBFT XBFZ  D FZ LABELEQSS2ENDALIGNFROM REFEQSS1 WE OBTAIN ZI  AXBFZ  BBF FZTHE MATRIX I IS THE IDENTITY MATRIX  THEN XBFZ  ZIA1 BBF FZWHERE ZIA1 IS THE MATRIX INVERSE OF ZIA  MATRIX INVERSESARE DISCUSSED IN CHAPTER REFCHAPMATINV  SUBSTITUTINGXBFZ INTO REFEQSS2 WE OBTAIN YZ  CBFT ZIA1 BBF  DFZSINCE YZ AND FZ ARE SCALAR SIGNALS WE CAN FORM THEIR RATIO TOOBTAIN THE SYSTEM FUNCTIONBEGINEQUATION HZ  FRACYZFZ  CBFT ZIA1 BBF  DLABELEQHZ3ENDEQUATIONBEGINEXAMPLE  WE WILL GO FROM A SYSTEM FUNCTION TO STATESPACE FORM AND BACK  LET HZ  FRAC3  2Z1  4 Z21  3Z1  5 Z2IN SOME LITERATURE IT IS COMMON TO ELIMINATE NEGATIVE POWERS OFZ IN THE SYSTEM FUNCTIONS  THIS CAN BE DONE BY MULTIPLYING BYZ2Z2 HZ  FRAC3Z2  2Z  4Z2  3Z  5PLACING THE SYSTEM IN CONTROLLER FORM WE HAVE BBF  BEGINBMATRIX0 1 ENDBMATRIX QQUAD CBF BEGINBMATRIX435  233 ENDBMATRIX  BEGINBMATRIX  11  7  ENDBMATRIX A  BEGINBMATRIX01  53 ENDBMATRIX QQUAD D3TO RETURN TO A TRANSFER FUNCTION WE FIRST COMPUTE ZIA  BEGINBMATRIXZ  1  5  Z3 ENDBMATRIXAND ZIA1  FRAC1ZZ3  5BEGINBMATRIX Z3  1  5   ZENDBMATRIXINDEXMATRIX INVERSEMATSIZE22THE INVERSE OF A MATSIZE22 MATRIX IS BOXEDBEGINBMATRIXA  B  CD ENDBMATRIX  FRAC1AD  BCBEGINBMATRIX D  B  C  A ENDBMATRIX THEN USING REFEQHZ3 WE OBTAIN HZ  FRAC1Z23Z5117BEGINBMATRIXZ3  1  5  ZENDBMATRIX BEGINBMATRIX0  1 ENDBMATRIX  D  FRAC3Z2  2Z   4Z23Z5AS EXPECTEDTO EMPHASIZE THAT THE STATESPACE REPRESENTATION IS NOT UNIQUE LET ATILDE  BEGINBMATRIX 5  45  15  35ENDBMATRIXQQUADBBFTILDE  BEGINBMATRIX 1  1 ENDBMATRIX QQUAD CBFTILDE BEGINBMATRIX 2  9 ENDBMATRIX QQUAD DTILDE  3THIS SYSTEM IS NOT IN CONTROLLER FORM  WE  MAY VERIFY THAT HTILDEZ  CBFTILDET ZIATILDE1 BBFTILDE  DTILDE  HZENDEXAMPLESUBSUBSECTIONSOLUTION OF THE STATESPACE DIFFERENCE EQUATIONIT CAN BE SHOWN SEE EXERCISE REFEXSTATEOUT THAT STARTING FROM ANINITIAL STATE XBF0 THE STATESPACE SYSTEM REFEQSTATE2 HASTHE SOLUTIONBEGINEQUATION XBFT  AT XBF0  SUMK0T1 AK BBF FT1KLABELEQXNDT1ENDEQUATIONTHE SUM IS SIMPLY THE CONVOLUTION OF AT BBF WITH FT1  THEOUTPUT IS YT  CBFT AT XBF0  SUMK0T1 CBFT AK BBFFT1K  D FTTHE QUANTITIES CBFT AK BBF ARE CALLED THE EM MARKOV  PARAMETERS INDEXMARKOV PARAMETERS OF THE SYSTEM THEY CORRESPONDTO THE IMPULSE RESPONSE OF THE SYSTEM ABBFCBFTSUBSUBSECTIONMULTIPLE INPUTS AND OUTPUTSSTATESPACE REPRESENTATION CAN BE USED TO REPRESENT SIGNALS WITHMULTIPLE INPUTS AND OUTPUTS  FOR EXAMPLE A SYSTEM MIGHT BE DESCRIBEDBY BEGINALIGNEDXBFT1  BEGINBMATRIX X1T1  X2T1  X3T1  ENDBMATRIX BEGINBMATRIX 321  125  211 ENDBMATRIXXBFT  BEGINBMATRIX21  15  11 ENDBMATRIX BEGINBMATRIX F1T  F2T ENDBMATRIX YBFT  BEGINBMATRIX Y1T   Y2T ENDBMATRIX BEGINBMATRIX2  4 6  120 ENDBMATRIXXBFTENDALIGNEDTHIS SYSTEM HAS THREE STATE VARIABLES TWO INPUTS AND TWO OUTPUTSIN GENERAL A MULTIINPUT MULTIOUTPUT SYSTEM IS OF THE FORMBEGINEQUATIONBEGINSPLITXBFT1   A XBFT  B UBFT YBFT  C XBFT  D UBFTENDSPLITLABELEQSTATEGENENDEQUATIONIF THERE ARE P STATE VARIABLES AND L INPUTS AND M OUTPUTS THEN BEGINALIGNEDA  TEXT IS  MATSIZEPP B  TEXT IS  MATSIZEPL C  TEXT IS  MATSIZEMP D  TEXT IS  MATSIZEMLENDALIGNEDSUBSUBSECTIONSTATESPACE SYSTEMS IN NOISEA SIGNAL MODEL THAT ARISES FREQUENTLY IN PRACTICE ISBEGINEQUATIONBEGINSPLITXBFT1  A XBFT  B UBFT  WBFT YBFT  C XBFT  D UBFT  VBFTENDSPLITLABELEQSTATEGEN1ENDEQUATIONTHE SIGNALS WBFT AND VBFT REPRESENT NOISE PRESENT IN THESYSTEM  THE VECTOR WBFT IS AN INPUT TO THE SYSTEM THATREPRESENTS UNKNOWN RANDOM COMPONENTS  FOR EXAMPLE IN MODELINGAIRPLANE DYNAMICS WBFT MIGHT REPRESENT RANDOM GUSTS OF WINDTHE VECTOR VBFT REPRESENTS MEASUREMENT NOISE  MEASUREMENT NOISEIS A FACT OF LIFE IN MOST PRACTICAL CIRCUMSTANCES  GETTING USEFULRESULTS OUT OF NOISY MEASUREMENTS IS AN IMPORTANT ASPECT OF SIGNALPROCESSING  IT HAS BEEN SAID THAT NOISE IS THE SIGNAL PROCESSORSBREAD AND BUTTER WITHOUT THE NOISE MANY PROBLEMS WOULD BE TOO TRIVIALTO BE OF SIGNIFICANT INTERESTTHIS BOOK WILL TOUCH ON SOME ASPECTS OF SYSTEMS IN STATESPACE FORMBUT A THOROUGH STUDY OF LINEAR SYSTEMS INCLUDING STATESPACECONCEPTS IS BEYOND THE SCOPE OF THIS BOOK  FOR SUPPLEMENTARYTREATMENTS SEE THE REFERENCE SECTION AT THE END OF THIS CHAPTERBEGINEXERCISESITEM PLACE THE FOLLOWING INTO STATE VARIABLE FORM CONTROLLER  CANONICAL FORM AND DRAW A REALIZATIONBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2ENDARRAYITEM DETERMINE THE FIRST FOUR NONZERO MARKOV PARAMETERS OF THE SYSTEMS IN THE  PREVIOUS EXERCISEITEM IN ADDITION TO THE BLOCK DIAGRAM SHOWN IN FIGURE  REFFIGTRANSFER2 THERE ARE MANY OTHER FORMS  THIS PROBLEM  INTRODUCES ONE OF THEM THE EM OBSERVER CANONICAL FORM INDEXOBSERVER CANONICAL FORM  BEGINENUMERATE  ITEM SHOW THAT THE ZTRANSFORM RELATION IMPLIED BY REFEQARMA    CAN BE WRITTEN ASBEGINEQUATIONBEGINSPLIT YZ   BBAR0 FZ  BBAR1 FZ  ABAR1 YZZ1  BBAR2 FZ  ABAR2 YZ Z2   CDOTS  BBARP FZ  ABARP YZZ1ENDSPLITLABELEQBLOCK2ENDEQUATIONITEM DRAW A BLOCK DIAGRAM REPRESENTING REFEQBLOCK2 CONTAINING  P DELAY ELEMENTS ITEM LABEL THE OUTPUTS OF THE DELAY ELEMENTS FROM RIGHT TO LEFT AS  X1 X2 LDOTS XP  SHOW THAT THE SYSTEM CAN BE PUT INTO STATE  SPACE FORM WITH A  BEGINBMATRIX ABAR1  1  0  CDOTS  0 ABAR2  0  1  CDOTS  0 VDOTS ABARP1  0  0 CDOTS  0 ABARP  0  0  CDOTS  1 ENDBMATRIXQQUAD BBF  BEGINBMATRIX BBAR1  ABAR1 BBAR0  BBAR2  ABAR2 BBAR0 CDOTS BBARP1  ABARP1 BBAR0 BBARP  ABARP BBAR0 ENDBMATRIXQQUAD CBF  BEGINBMATRIX 1  0  0  VDOTS  0 ENDBMATRIXQQUAD D  BBAR0A MATRIX A OF THIS FORM IS SAID TO BE IN EM SECOND COMPANION  FORM INDEXSECOND COMPANION FORMITEM DRAW THE BLOCK DIAGRAM IN OBSERVER CANONICAL FORM FOR HZ  FRAC 2  3Z1  4 Z21  Z1  6Z2  7 Z3AND DETERMINE THE SYSTEM MATRICES ABBF CBFT DENDENUMERATEITEM ANOTHER BLOCK DIAGRAM REPRESENTATION IS BASED UPON THE PARTIAL  FRACTION EXPANSION    BEGINENUMERATE  ITEM ASSUME INITIALLY THAT THERE ARE NO REPEATED ROOTS SO THAT HZ  SUMK1P FRACNK1PK Z1ITEM DRAW A BLOCK DIAGRAM REPRESENTING THE PARTIAL FRACTION EXPANSION  BY USING THE FACT THAT FRACYZFZ  FRAC11P Z1 HAS THE BLOCK DIAGRAM BEGINCENTERINPUTPICTUREDIRTRANSFER5LATEXENDCENTERITEM LET XI I12LDOTS P DENOTE THE OUTPUTS OF THE DELAY  ELEMENTS  SHOW THAT THE SYSTEM CAN BE PUT INTO STATESPACE FORM  WITH  A  BEGINBMATRIX P1  0  0  CDOTS 00P2  0  CDOTS  0  VDOTS 0  0  0  CDOTS  PP ENDBMATRIXQQUAD BBF  BEGINBMATRIX 1  1  VDOTS  1 ENDBMATRIXQQUAD CBF  BEGINBMATRIX  N1  N2  VDOTS  NP ENDBMATRIXQQUAD D  B0A MATRIX A IN THIS FORM IS SAID TO BE A EM DIAGONALMATRIX INDEXDIAGONAL MATRIXITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF  HZ  FRAC 1  2Z1 1  5 Z1  6 Z2AND DRAW THE BLOCK DIAGRAM BASED UPON IT  DETERMINE ABBF CBFDITEM WHEN THERE ARE REPEATED ROOTS THINGS ARE SLIGHTLY MORE  COMPLICATED  CONSIDER FOR SIMPLICITY A ROOT APPEARING ONLY  TWICE  DETERMINE THE PARTIAL FRACTION EXPANSION OF HZ  FRAC1Z11 2 Z115Z12BE CAREFUL ABOUT THE REPEATED ROOT  ITEM DRAW THE BLOCK DIAGRAM CORRESPONDING TO HZ IN PARTIAL  FRACTION FORM USING ONLY THREE DELAY ELEMENTS  ITEM SHOW THAT THE STATE VARIABLES CAN BE CHOSEN SO THAT A  BEGINBMATRIX 5 00 15  0 0  0  2  ENDBMATRIXA MATRIX IN THIS FORM BLOCKS ALONG THE DIAGONAL EACH BLOCK BEINGEITHER DIAGONAL OR DIAGONAL WITH ONES IN IT AS SHOWN IS IN EM JORDANFORM INDEXJORDAN FORMENDENUMERATEITEM SHOW THAT THE SYSTEM IN REFEQSTATE3 HAS THE SAME TRANSFER  FUNCTION AND SOLUTION AS DOES THE SYSTEM IN REFEQSTATE2ITEM LABELEXSTATEOUT SHOW THAT REFEQXNDT1 IS CORRECTITEM FOR A SYSTEM IN STATESPACE REPRESENTATION  BEGINENUMERATE  ITEM SHOW THAT REFEQXNDT1 IS CORRECT  ITEM FOR A TIMEVARYING SYSTEM AS IN REFEQSTATE4 DETERMINE    A REPRESENTATION SIMILAR TO REFEQXNDT1  ENDENUMERATEITEM CITEKAILATH80 LET A1BBF1CBF1T AND  A2BBF2CBF2T BE TWO SYSTEMS  DETERMINE THE SYSTEM  ABBFCBFT OBTAINED BY CONNECTING THESE  BEGINENUMERATE  ITEM IN SERIES  ITEM IN PARALLEL  ITEM IN A FEEDBACK CONFIGURATION WITH A1BBF1CBF1T IN    THE FORWARD LOOP AND A2BBF2CBF2T IN THE FEEDBACK LOOP  ENDENUMERATEITEM SHOW THAT  BEGINBMATRIX A A1  0  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  ZEROBF ENDBMATRIX QQUAD CBFT QBFTAND BEGINBMATRIX A 0  A1  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  QBF ENDBMATRIX QQUAD CBFT ZEROBFAND ABBFCBFT ALL HAVE THE SAME TRANSFER FUNCTION FOR ALLVALUES OF A1 A2 AND QBF THAT LEAD TO VALID MATRIXOPERATIONS  CONCLUDE THAT REALIZATIONS CAN HAVE DIFFERENT NUMBERS OF STATESENDEXERCISESSUBSECTIONCONTINUOUSTIME NOTATIONLABELSECCONTSTATEFOR CONTINUOUSTIME SIGNALS AND SYSTEMS THE CONCEPTS FOR INPUTOUTPUTRELATIONS TRANSFER FUNCTIONS AND STATESPACE REPRESENTATIONSTRANSLATE DIRECTLY WITH Z1 UNIT DELAY REPLACED BY1S INTEGRATION  THE READER IS ENCOURAGED TO REVIEW THEDISCRETETIME NOTATIONS PRESENTED ABOVE AND REFORMULATE THEEXPRESSIONS GIVEN IN TERMS OF CONTINUOUSTIME SIGNALS  THE PRINCIPALDIFFERENCE BETWEEN DISCRETE TIME AND CONTINUOUS TIME ARISES IN THEEXPLICIT SOLUTION OF THE DIFFERENTIAL EQUATIONBEGINEQUATIONBEGINSPLITXBFDOTT  AT XBFT  BT FBFT YBFT  CT XBF  DT FBFTENDSPLIT  LABELEQXNCT1ENDEQUATIONFOR THE TIMEINVARIANT SYSTEM WHEN ABCD IS CONSTANT THESOLUTION ISBEGINEQUATION XBFT  EAT XBF0  INT0 T EATLAMBDA BFBFLAMBDA DLAMBDALABELEQXBFT2ENDEQUATIONWHERE EAT IS THE EM MATRIX EXPONENTIAL INDEXMATRIX EXPONENTIAL DEFINED IN TERMS OF ITS  TAYLOR SERIES INDEXTAYLOR SERIESBEGINEQUATION EAT  I  AT  A2 FRACT22  A3 FRACT33  CDOTS LABELEQEXPMAT1ENDEQUATIONWHERE I IS THE EM IDENTITY MATRIX  WE NOTE IN PARTICULAR THAT FRACDDT EAT  AEATSEE SECTION REFSECTAYLOR FOR A REVIEW OF TAYLOR SERIES ANDSECTION REFSECDIAGONAL FOR MORE ON THE MATRIX EXPONENTIAL  THEMATRIX EXPONENTIAL CAN ALSO BE EXPRESSED IN TERMS OF LAPLACETRANSFORMS INDEXMATRIX EXPONENTIALSEESTATE TRANSITION MATRIX EAT  LC1SIA1WHERE SIA IS KNOWN AS THE EM CHARACTERISTIC MATRIX OF A ANDLC CDOT DENOTES THE LAPLACE TRANSFORM OPERATOR INDEXLLCINDEXLAPLACE TRANSFORM LCFT  INT0INFTY FTESTDTAN INTERESTING AND FRUITFUL CONNECTION IS THE FOLLOWING  RECALL THEGEOMETRIC EXPANSIONBEGINEQUATION FRAC11X  1XX2  X3  CDOTSLABELEQGEOM1ENDEQUATIONWHICH CONVERGES FOR X  1  INDEXGEOMETRIC SERIES THIS ALSOAPPLIES TO GENERAL OPERATORS INCLUDING MATRICES SO THAT FOR ANOPERATOR FBEGINEQUATION  IF1  I  F  F2  F3  CDOTSLABELEQNEUMANN1ENDEQUATIONWHEN F1  THE NOTATION F SIGNIFIES THE OPERATOR NORM ITIS DISCUSSED IN SECTION REFSECMATNORM  THE EXPANSIONREFEQNEUMANN1 IS KNOWN AS THE NEUMANN EXPANSION SEE SECTIONREFSECNEUM  INDEXNEUMANN EXPANSION USING REFEQNEUMANN1THE EXPRESSION SIA1 IS FRAC1SI  AS  A2S2  CDOTSFROM WHICH THE TAYLOR SERIES FORMULA REFEQEXPMAT1 FOLLOWSIMMEDIATELY USING THE INVERSE LAPLACE TRANSFORMFOR THE TIMEINVARIANT SINGLEINPUT SINGLEOUTPUT SYSTEM BEGINALIGNEDXBFDOTT  A XBFT  BBF FT YT  CBFT XBFTENDALIGNEDTHE TRANSFER FUNCTION IS HS  CBFSIA1 BBFUSING REFEQNEUMANN1 WE WRITE HS  SUMI1INFTY HI SIWHERE HI  CBFT AI1 BBF ARE THE MARKOV PARAMETERS OF THECONTINUOUSTIME SYSTEM INDEXMARKOV PARAMETERSTHE FIRST TERM OF REFEQXBFT2 IS THE SOLUTION OF THE HOMOGENEOUSDIFFERENTIAL EQUATION XBFDOTT  A XBFTWHILE THE SECOND TERM OF REFEQXBFT2 IS THE PARTICULAR SOLUTIONOF  XBFDOT  AT XBFT  BT FBFTIT IS STRAIGHTFORWARD TO SHOW SEE EXERCISE REFEXUPDATEDEQ THATSTARTING FROM A STATE XBFTAU THE STATE AT TIME T CAN BEDETERMINED ASBEGINEQUATION XBFT  EATTAU XBFTAU  INTTAUT EATTAU BFBFLAMBDA DLAMBDALABELEQSTATEUPDATEENDEQUATIONSINCE EATTAU PROVIDES THE MECHANISM FOR MOVING FROM STATEXBFTAU TO STATE XBFT IT IS CALLED THE EM STATE  TRANSITION MATRIX LABELSTATE TRANSITION MATRIXFOR THE TIMEVARYING SYSTEM REFEQXNCT1 INDEXTIMEVARYING SYSTEM THE SOLUTION CAN BE WRITTEN ASBEGINEQUATIONXBFT  PHIT0 XBF0  INT0T PHITLAMBDA BLAMBDAFBFLAMBDA DLAMBDALABELEQXBFT3ENDEQUATIONWHERE PHITTAU IS THE STATETRANSITION MATRIX INDEXSTATE  TRANSITION MATRIX  NOT DETERMINED BYTHE MATRIX EXPONENTIAL IN THE TIMEVARYING CASE  THE FUNCTIONPHITTAU HAS THE FOLLOWING PROPERTIESBEGINENUMERATEITEM PHITT  IITEM PARTIALDPHITTAUT  AT PHITTAUITEM PHITTAU  PHITAUT1 THE MATRIX INVERSEENDENUMERATESUBSUBSECTIONCONTINUOUSTIME NOTATIONWE NOW SUMMARIZE BRIEFLY SOME SYSTEMS CONCEPTS FOR CONTINUOUS TIMETHE KEY DIFFERENCE IS THAT INSTEAD OF DIFFERENCE EQUATIONS ANDZTRANSFORMS WE DEAL WITH DIFFERENTIAL EQUATIONS AND LAPLACETRANSFORMS  WE WILL DEAL INPUTOUTPUT RELATIONSHIPS OF THE FORM YT  A1 FRACDDTYT  A2 FRACD2DT2 YT  CDOTS AP FRACDPDTP YT  B0 UT B1 FRACDDT UT  CDOTS  BQ FRACDQDTQ UTTAKING THE LAPLACE TRANSFORM AGAIN SETTING INITIAL CONDITIONS TOZERO YS SUMK0P AK SK  UZ SUMK0Q BK SKWHICH WE WRITE AS YS AS  US BSTHE SYSTEM FUNCTION ISBEGINEQUATION HS  FRACYSUS  FRACSUMK0Q BK SKSUMK0P AK SK   FRACSUMK0Q BK SK1 SUMK1PAK SK  FRACBSAS LABELEQHS1ENDEQUATIONTHE OUTPUT CAN BE EXPRESSED IN TERM OF THE INPUT AS YS  HSUSTAKING THE INVERSE  LAPLACE TRANSFORM AND RECALLING THECONVOLUTION PROPERTY MULTIPLICATION IN THE TRANSFORM DOMAINCORRESPONDS TO CONVOLUTION IN THE TIME DOMAIN WE OBTAIN YT  INTINFTYINFTY UTAU HTTAU DTAUWHERE THE IMPULSE RESPONSE HT IS THE INVERSE LAPLACE TRANSFORM OFHS  IN COMPUTING THE INVERSE TRANSFORM THE NUMERATOR ANDDENOMINATOR POLYNOMIALS OF THE SYSTEM FUNCTION HS OFREFEQHS1 ARE FACTORED HS  FRACB0 PRODK1Q SZIA0 PRODK1P SPIWHERE THE ZI ARE THE NONZERO ZEROS OF BS AND THE PI ARE THENONZERO ZEROS OF AS  IN THIS FORMWE OBSERVE THAT IF A POLE IS EQUAL TO A ZERO THE FACTORS CAN BECANCELED OUT OF BOTH THE NUMERATOR AND DENOMINATOR TO OBTAIN ANEQUIVALENT TRANSFER FUNCTION  ASSUMING FOR SIMPLICITY OF DISCUSSIONTHAT THE POLES ARE ALL UNIQUE NO REPEATED POLES AND THAT QP THENBY PARTIAL FRACTION EXPANSION THE SYSTEM FUNCTION CAN BE EXPRESSED ASBEGINEQUATIONHZ  SUMK1P FRACNKSPKLABELEQHS2ENDEQUATIONWHERE NK  HZSPKBIGLSPKTAKING THE CAUSAL INVERSE LAPLACE TRANSFORM OF REFEQHS2 WEOBTAIN HT  SUMK1P NK EPK TQQUAD T GEQ 0THE FUNCTIONS EPK T ARE THE NATURAL MODES OF THE SYSTEMFOR THE MODES TO BE BOUNDED IN TIME WE MUST HAVE REALPK LEQ 0IF THERE ARE REPEATED POLES IN HS A PARTIAL FRACTION CAN STILL BEOBTAINED BUT SOMEWHAT MORE CARE IS REQUIRED  IF HZ  FRACBSSPRTHEN THE PFE IS HZ  FRACK0SPR  FRACK1SPR1  CDOTS FRACKR1SPWHEREBEGINEQUATION  LABELPFES  KJ  FRAC1J FRACDJDSJ SPR HS BIGLSPENDEQUATIONIF Q GEQ P THEN THE RATIO OF POLYNOMIALS IS FIRST DIVIDED OUTBLOCK DIAGRAMS FOR CONTINUOUSTIME TRANSFER FUNCTIONS CAN BE DERIVEDJUST AS FOR DISCRETETIME TRANSFER FUNCTIONS  FIGUREREFFIGTRANSFER4 SHOWS THE CONTROLLER CANONICAL FORM OF A BLOCKDIAGRAM WITH STATEVARIABLE LABELS ON THE OUTPUTS OF THE INTEGRATORSBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRTRANSFER4LATEX    CAPTIONCONTROLLER CANONICAL FORM FOR A CONTINUOUS TIME SYSTEM    LABELFIGTRANSFER4  ENDCENTERENDFIGUREFROM THE DIAGRAM WE CAN READ OFF THE STATEVARIABLE EQUATIONSBEGINEQUATIONBEGINSPLITXDOT1T  X2T XDOT2T  X3T  VDOTS XDOTP1T  XPT XDOTPT  UT A1 XPT A2 XP1T  CDOTS  AP1 X2T AP X1T SMALLSKIP YT  BP X1T  BP1 X2T  CDOTS  B2 XP1T  B1XPT    QQUAD B0UT  A1 XPT  A2 XP1T  CDOTS  AP X1TENDSPLITLABELEQSTATE3ENDEQUATIONTHE STATE VECTOR IS XBFT  BEGINBMATRIX X1T  X2T  VDOTS  XPTENDBMATRIX IN STATEVARIABLE FORM WE HAVEBEGINEQUATIONBEGINSPLITXBFDOTT  A XBFT  BBF UT YT  CBFT XBFT  D UTENDSPLITLABELEQSTATE4ENDEQUATIONWHERE XBFDOTT MEANS TO TAKE THE TIME DERIVATIVE OF EACH COMPONENTOF XBFT SEPARATELY AND A BBF CBF AND D ARE ASBEFORE  THE TRANSFER FUNCTION HS CAN BE EXPRESSED IN TERMS OF THE SYSTEMABBF CBF D ASBEGINEQUATION HS  FRACYSUS  CBFT SIA1 BBF  DLABELEQHS3ENDEQUATIONTHE OUTPUT CAN BE EXPRESSED AS YT  CBFT EATXBF0  INT0T CBFT EATTAU BUTAU  DTAUWHERE EAT IS THE MATRIX EXPONENTIAL DEFINED IN TERMS OF ITSTAYLOR SERIES EX  I  X  FRACX22  FRACX33  CDOTSFOR ANY SQUARE MATRIX X  TAYLOR SERIES ARE REVIEWED IN SECTIONREFSECTAYLOR  THE DYNAMICAL PROPERTIES OF THE MATRIX EXPONENTIALARE DISCUSSED IN SECTION REFSECMATEXPMORE GENERALLY WITH MULTIPLE INPUTS AND MULTIPLE OUTPUTS AND IN THEPRESENCE OF NOISE WE HAVE HAVE BEGINALIGNED XBFDOTT  A XBFT  B UBFT   WBFT YBFT  C XBFT  D UBFT  VBFTENDALIGNEDBEGINEXERCISESITEM FOR THE SYSTEM FUNCTION HZ  FRACS3  6S2  11 S  6 S3  9S2  23 S  15BEGINENUMERATEITEM DRAW THE CONTROLLER CANONICAL BLOCK DIAGRAMITEM DRAW THE BLOCK DIAGRAM IN JORDAN FORM DIAGONAL  FORM INDEXJORDAN FORMITEM HOW MANY MODES ARE REALLY PRESENT IN THE SYSTEM  THE PROBLEM  HERE IS THAT A EM MINIMAL REALIZATION OF A IS NOT OBTAINED  DIRECTLY FROM THE HZ AS GIVENENDENUMERATEITEM CITEKAILATH80 IF ABBFCBFTD WITH D NEQ 0  DESCRIBES A SYSTEM HS IN STATESPACE FORM SHOW THAT A  BBF CBFTD BBFD CBFTD 1DDESCRIBES A SYSTEM WITH SYSTEM FUNCTION 1HSITEM LABELEXUPDATEDEQ STATESPACE SOLUTIONS  BEGINENUMERATE  ITEM SHOW THAT REFEQXBFT2 IS A SOLUTION TO THE DIFFERENTIAL  EQUATION IN REFEQXNCT1 FOR CONSTANT ABCDITEM SHOW THAT AN UPDATE FROM XBFTAU TO XBFT IS AS GIVEN  IN REFEQSTATEUPDATEITEM SHOW THAT REFEQXBFT3 IS A SOLUTION TO THE DIFFERENTIAL  EQUATION IN REFEQXNCT1 FOR NONCONSTANT ABCD PROVIDED  THAT PHI SATISFIES THE PROPERTIES GIVEN  ENDENUMERATEITEM FIND A SOLUTION TO THE DIFFERENTIAL EQUATION DESCRIBED BY THE  STATESPACE EQUATIONSBEGINALIGNED XBFDOTT  BEGINBMATRIX 01  10 ENDBMATRIX XBFT YT  1 0 XBFTENDALIGNEDWITH XBF0  XBF0  THESE EQUATIONS DESCRIBE SIMPLE HARMONICMOTIONITEM FOR THE SYSTEM DESCRIBED BYBEGINALIGNED XBFDOTT  BEGINBMATRIX 2  0  1  1 ENDBMATRIX XBFT BEGINBMATRIX 2  1 ENDBMATRIX FT YT  0  2 XBFTENDALIGNEDBEGINENUMERATEITEM DETERMINE THE TRANSFER FUNCTION HSITEM FIND THE PARTIAL FRACTION EXPANSION OF HSENDENUMERATEITEM VERIFY REFEQGEOM1 BY LONG DIVISION LABELGEOMETRIC    SERIESITEM ENDEXERCISESSUBSECTIONISSUES AND APPLICATIONSTHE NOTATION INTRODUCED IN THE PREVIOUS SECTIONS ALLOWS US NOW TODISCUSS A VARIETY OF ISSUES OF BOTH PRACTICAL AND THEORETICALIMPORTANCE  HERE ARE A FEW EXAMPLESBEGINITEMIZEITEM GIVEN A DESIRED FREQUENCY RESPONSE SPECIFICATION   EITHER HEJOMEGAFOR DISCRETETIME SYSTEMS OR HJOMEGAFOR CONTINUOUSTIME SYSTEMS DETERMINE THE COEFFICIENTS AIAND BI TO MEET OR CLOSELY APPROXIMATE THE RESPONSESPECIFICATION  THIS IS THE EM FILTER DESIGN PROBLEM INDEXFILTER  DESIGNITEM GIVEN A SEQUENCE OF OUTPUT DATA FROM A SYSTEM HOW CAN THE  PARAMETERS OF THE SYSTEM BE DETERMINED IF THE INPUT SIGNAL IS KNOWN  IF THE INPUT SIGNAL IS NOT KNOWNITEM DETERMINE  A MINIMAL REPRESENTATION OF A SYSTEMITEM GIVEN A SIGNAL OUTPUT FROM A SYSTEM DETERMINE A PREDICTOR FOR  THE SIGNAL INDEXLINEAR PREDICTORITEM DETERMINE A MEANS OF EFFICIENTLY CODING REPRESENTING A SIGNAL  MODELED AS THE OUTPUT OF AN LTI SYSTEMITEM DETERMINE THE SPECTRUM OF THE OUTPUT OF AN LTI SYSTEMITEM DETERMINE THE MODES OF A SYSTEMITEM FOR ALGORITHMS OF THE SORT JUST PRESCRIBED DEVELOP  COMPUTATIONALLY EFFICIENT ALGORITHMSITEM SUPPOSE THE MODES OF A SIGNAL ARE NOT WHAT WE WANT THEM TO BE  DEVELOP A MEANS OF USING FEEDBACK TO BEND THEM TO SUIT OUR PURPOSESENDITEMIZEEXAMINATION OF MANY OF THESE ISSUES IS TAKEN UP AT APPROPRIATE PLACESTHROUGHOUT THIS BOOK WITH VARYING DEGREES OF COMPLETENESSSUBSUBSECTIONESTIMATION OF PARAMETERS LINEAR PREDICTIONINDEXLINEAR PREDICTORIT MAY OCCUR THAT A SIGNAL CAN BE MODELED AS THE OUTPUT OF ADISCRETETIME SYSTEM WITH SYSTEM FUNCTION HZ FOR WHICH THEPARAMETERS PQ B0LDOTS BQ A1 LDOTS AP ARE NOT KNOWN  GIVEN A SEQUENCEOF OBSERVATIONS Y0Y1LDOTS WE WANT TO DETERMINE IF POSSIBLETHE PARAMETERS OF THE SYSTEM  THIS BASIC PROBLEM HAS TWO MAJORVARIATIONSBEGINITEMIZEITEM THE INPUT FT IS DETERMINISTIC AND KNOWNITEM THE INPUT FT IS RANDOMENDITEMIZEOTHER COMPLICATIONS MAY ALSO BE MODELED IN PRACTICE  FOR EXAMPLE ITMAY BE THAT THE OUTPUT YT IS CORRUPTED BY NOISE SO THAT THE DATAAVAILABLE IS ZT  YT  WTWHERE WT IS A NOISE OR ERROR SIGNAL  THIS IS A SIGNAL PLUSNOISE MODEL THAT WE WILL EMPLOY FREQUENTLY INDEXSIGNAL PLUS  NOISEIN THE CASE WHERE THE INPUT IS KNOWN AND THERE IS NEGLIGIBLE OR NOMEASUREMENT NOISE IT IS STRAIGHTFORWARD TO SET UP A SYSTEM OF LINEAREQUATIONS TO DETERMINE THE SYSTEM PARAMETERS  FOR THE ARMAPQSYSTEM OF REFEQARMA IF THE ORDER PQ IS KNOWN A SYSTEM OFEQUATIONS TO FIND THE UNKNOWN PARAMETERS CAN BE SET UP ASBEGINEQUATIONLABELEQARMAIDAXBF  BBFENDEQUATIONIN WHICH A  BEGINBMATRIX YP1  YP2  CDOTS  Y0  FP  FP1   CDOTS  FPQ1 YP  YP1  CDOTS  Y1  FP1  FP1  CDOTS  FPQ  VDOTS  YN1  YN2  CDOTS  YNP  FN  FN1  CDOTS  FNQ1 ENDBMATRIX XBF  BEGINBMATRIX ABAR1  ABAR2  VDOTS  ABARP  BBAR0   BBAR1  VDOTS  BBARQENDBMATRIXQQUAD TEXTANDQQUADBBF   BEGINBMATRIX YP  YP1  VDOTS  YNENDBMATRIXWHERE N IS LARGE ENOUGH THAT THERE ARE AS MANY EQUATIONS ASUNKNOWNS  WHEN THERE IS MEASUREMENT NOISE IN THE SYSTEM N CAN BEINCREASED SO THAT THERE ARE MORE EQUATIONS THAN UNKNOWNS AND ALEASTSQUARES SOLUTION CAN BE COMPUTED AS DISCUSSED IN CHAPTERSREFCHAPVECTAP AND REFCHAPMATFACT  AN IMPORTANT SPECIAL CASE IN THIS PARAMETER ESTIMATION PROBLEM INWHICH THE INPUT IS ASSUMED TO BE NOISE AND WHEN HZ IS KNOWN TO BE OR ASSUMED TO BE  AN ARP SYSTEM WITH P KNOWN HZ  FRAC11SUMK1P AK ZKSUCH A MODEL IS COMMONLY ASSUMED IN SPEECH PROCESSING INDEXSPEECH  PROCESSING WHERE A SPEECH SIGNAL IS MODELED AS THE OUTPUT OF ANALLPOLE SYSTEM DRIVEN BY EITHER A ZEROMEAN UNCORRELATED SIGNAL INTHE CASE OF UNVOICED SPEECH SUCH AS THE LETTER S OR BY APERIODIC PULSE SEQUENCE IN THE CASE OF VOICED SPEECH SUCH AS THELETTER A  WE ASSUME THAT THE SIGNAL IS GENERATED ACCORDING TO YT  ABFT YBFT1  FTFURTHER ASSUMING HERE THE MODEL USES REAL DATA  OUR ESTIMATED MODELHAS OUTPUT YHATT WHERE YHATT  ABFHATT YBFTAND ABFHAT  BEGINBMATRIX AHAT1  AHAT2  VDOTS  AHATPENDBMATRIXTHE MARK HAT INDEXHAT ON A QUANTITY INDICATES ANESTIMATED OR APPROXIMATE VALUE  WE CAN INTERPRET THE ESTIMATED ARSYSTEM AS A EM LINEAR PREDICTOR THE VALUE YHATT IS THEPREDICTION OF YT GIVEN THE PAST DATA YT1 YT2LDOTSYTP  THE PREDICTION PROBLEM CAN BE STATED AS FOLLOWS DETERMINETHE PARAMETERS AHAT1LDOTSAHATP TO GET THE BESTPREDICTION  THERE IS AN ERROR BETWEEN WHAT IS ACTUALLY PRODUCED BYTHE SYSTEM AND THE PREDICTED VALUE ET  YT  YHATTTHIS IS ILLUSTRATED IN FIGURE REFFIGPREDICT1  A GOODPREDICTOR WILL MAKE THE ERROR AS SMALL AS POSSIBLE IN SOME SENSETHE SOLUTION TO THE PREDICTION PROBLEM IS DISCUSSED IN CHAPTERREFCHAPVECTAPBEGINFIGUREHTBP  CENTERLINEINPUTPICTUREDIRPREDICT1LATEX  CENTERLINEINPUTPICTUREDIRPREDICT1  CAPTIONPREDICTION ERROR   LABELFIGPREDICT1ENDFIGUREONE APPLICATION OF LINEAR PREDICTION IS TO DATA COMPRESSIONINDEXDATA COMPRESSION WE DESIRE TO REPRESENT A SEQUENCE OF DATAUSING THE SMALLEST NUMBER OF BITS POSSIBLE  IF THE SEQUENCE WERECOMPLETELY DETERMINISTIC SO THAT YT IS A DETERMINISTIC FUNCTIONOF PRIOR OUTPUTS WE WOULD NOT NEED TO SEND ANY BITS TO DETERMINEYT IF THE PRIOR OUTPUTS WERE KNOWN WE COULD SIMPLY USE A PERFECTPREDICTOR TO REPRODUCE THE SEQUENCE  IF YT IS NOT DETERMINISTICWE PREDICT YT THEN CODE QUANTIZE ONLY THE PREDICTION ERROR  IFTHE PREDICTION ERROR IS SMALL THEN ONLY A FEW BITS ARE REQUIRED TOACCURATELY REPRESENT IT  CODING IN THIS WAY IS CALLED DIFFERENTIALPULSE CODE MODULATION  WHEN PARTICULAR FOCUS IS GIVEN TO THE PROCESSOF DETERMINING THE PARAMETERS ABFHAT IT MAY BE CALLED LINEARPREDICTIVE CODING LPC  TO BE SUCCESSFUL IT MUST BE POSSIBLE TODETERMINE THE COEFFICIENTS INSIDE THE PREDICTORLINEAR PREDICTION ALSO HAS APPLICATIONS TO PATTERN RECOGNITIONSUPPOSE THERE ARE SEVERAL CLASSES OF SIGNALS TO BE DISTINGUISHED FOREXAMPLE SEVERAL SPEECH SOUNDS TO BE RECOGNIZED  EACH SIGNAL WILLHAVE ITS OWN SET OF PREDICTION COEFFICIENTS SIGNAL 1 HAS ABF1SIGNAL 2 HAS ABF2 AND SO FORTH  AN UNKNOWN INPUT SIGNAL CAN BEREDUCED BY ESTIMATING THE PREDICTION COEFFICIENTS THAT REPRESENT ITTO A VECTOR ABF  THEN ABF CAN BE COMPARED WITH ABF1ABF2 AND SO FORTH USING AN APPROPRIATE COMPARISON FUNCTION TODETERMINE WHICH SIGNAL THE UNKNOWN INPUT IS MOST SIMILAR TOWE CAN EXAMINE THE LINEAR PREDICTION PROBLEM FROM ANOTHER PERSPECTIVEIF YZ  HZFZTHEN FZ  YZ FRAC1HZTHAT IS FT  YT  ABFT YBFT1IF WE REGARD YT AS THE INPUT THEN FT IS THE OUTPUT OF ANINVERSE SYSTEM  IF WE HAVE AN ESTIMATED SYSTEMHHATZ  FRAC11  SUMK1P AHATK ZKTHEN THE OUTPUT FHATT  YT  ABFHATT YBFT1SHOULD BE CLOSE IN SOME SENSE TO FT  A BLOCK DIAGRAM IS SHOWNIN FIGURE REFFIGINVLP  IN THIS CASE WE WOULD WANT TO CHOOSE THEPARAMETERS ABFHAT TO MINIMIZE IN SOME SENSE THE ERROR FT FHATT  THAT IS WE WANT TO DETERMINE A GOOD INVERSE FILTER FORHZBEGINFIGUREHTBP  CENTERLINEINPUTPICTUREDIRPREDICT2LATEX  CENTERLINEINPUTPICTUREDIRPREDICT2  CAPTIONLINEAR PREDICTOR AS AN INVERSE SYSTEM  LABELFIGINVLPENDFIGUREINTERESTINGLY USING EITHER THE POINT OF VIEW OF FINDING A GOODPREDICTOR OR FINDING A GOOD INVERSE FILTER PRODUCES THE SAME ESTIMATEIT IS ALSO INTERESTING IS THAT COMPUTATIONALLY EFFICIENT ALGORITHMSEXIST FOR SOLVING THE EQUATIONS THAT ARISE IN THE LINEAR PREDICTIONPROBLEM THESE ARE DISCUSSED IN CHAPTER REFCHAPSPECIALMATSUBSUBSECTIONESTIMATION OF PARAMETERS SPECTRUM ANALYSISINDEXSPECTRUM ANALYSISIT IS COMMON IN SIGNAL ANALYSIS TO CONSIDER THAT A GENERAL SIGNAL ISCOMPOSED OF SINUSOIDAL SIGNALS ADDED TOGETHER  DETERMINING THESEFREQUENCY COMPONENTS BASED UPON MEASURED SIGNALS IS CALLED EMSPECTRUM ESTIMATION OR EM SPECTRAL ANALYSIS  THERE ARE TWOGENERAL APPROACHES TO SPECTRAL ANALYSIS  THE FIRST APPROACH IS BYMEANS OF FOURIER TRANSFORMS IN PARTICULAR THE DISCRETE FOURIERTRANSFORM  THIS APPROACH IS CALLED NONPARAMETRIC SPECTRUMESTIMATION  THE SECOND APPROACH IS A PARAMETRIC APPROACH IN WHICH AMODEL FOR THE SIGNAL IS PROPOSED SUCH AS THE ONE IN REFEQARMAAND THEN THE PARAMETERS ARE ESTIMATED FROM THE MEASURED DATA  ONCETHESE ARE KNOWN THE SPECTRUM OF THE SIGNAL CAN BE DETERMINEDPROVIDED THAT THE MODELING ASSUMPTIONS ARE ACCURATE IT IS POSSIBLE TOOBTAIN BETTER SPECTRAL RESOLUTION WITH FEWER PARAMETERS USINGPARAMETRIC METHODSDISCUSSION OF SPECTRUM ANALYSIS REQUIRES SOME FAMILIARITY WITH THECONCEPTS OF ENERGY AND POWER SPECTRAL DENSITIES  INDEXENERGY  SPECTRAL DENSITY INDEXPOWER SPECTRAL DENSITYINDEXDISCRETETIME FOURIER TRANSFORM FOR A DISCRETETIMEDETERMINISTIC SIGNAL YT THE DISCRETETIME FOURIER TRANSFORM DTFT IS BOXEDYOMEGA  SUMTINFTYINFTY YT EJOMEGA T WHERE JSQRT1  THE EM ENERGY SPECTRAL DENSITY ESD IS AMEASURE OF HOW MUCH ENERGY THERE IS AT EACH FREQUENCY  AN EM ENERGY  SIGNAL YT HAS FINITE ENERGY INDEXENERGY SIGNAL SUMTINFTYINFTY YT2  INFTYFOR A DETERMINISTIC ENERGY SIGNAL THE ESD IS DEFINED BY GYYOMEGA  YOMEGA2WHERE THE SUBSCRIPT Y ON GYY INDICATES THE SIGNAL WHOSE ESD ISREPRESENTED  THE EM AUTOCORRELATION FUNCTION OF A DETERMINISTICSEQUENCE IS RHOYYK  SUMTINFTYINFTY YT YBARTKINDEXAUTOCORRELATION FUNCTIONTHEN SEE EXERCISE REFEXESD1BEGINEQUATIONGYYOMEGA  SUMKINFTYINFTY RHOYYK EJOMEGA KLABELEQESD1ENDEQUATIONTHAT IS THE ENERGY SPECTRAL DENSITY IS THE DTFT OF THEAUTOCORRELATION FUNCTIONTHE POWER SPECTRAL DENSITY PSD IS EMPLOYED FOR SPECTRAL ANALYSIS OFSTOCHASTIC SIGNALS  IT PROVIDES AN INDICATION OF HOW MUCH AVERAGEPOWER THERE IS IN THE SIGNAL AS A FUNCTION OF FREQUENCY  WE ASSUMETHAT THE SIGNAL IS ZERO MEAN EYT  0  FOR THE SIGNAL YTWITH AUTOCORRELATION FUNCTION RYYK WE ALSO ASSUME THAT THEAUTOCORRELATION DROPS OFF SUFFICIENTLY FAST THATBEGINEQUATION LIMN RIGHTARROW INFTY FRAC1N SUMKNN K RYYK  0LABELEQPSDDECENDEQUATIONTHE PSD IS DEFINED AS SYYOMEGA  SUMKINFTYINFTY RYYK EJOMEGA KTHAT IS THE PSD IS THE DTFT OF THE AUTOCORRELATION SEQUENCE ONE OF THE IMPORTANT PROPERTIES OF THE PSD IS THAT SYYOMEGA GEQ 0 QQUAD TEXTFOR ALL  OMEGATHIS CORRESPONDS TO THE PHYSICAL FACT THAT REAL POWER CANNOT BENEGATIVEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODECENTERLINEINPUTPICTUREDIRSYST2LATEXCENTERLINEINPUTPICTUREDIRSYST2    CAPTIONPSD INPUT AND OUTPUT    LABELFIGSYST2  ENDCENTERENDFIGUREA SIGNAL FT WITH PSD SFOMEGA INPUT TO A SYSTEM WITH SYSTEMFUNCTION HZ PRODUCES THE SIGNAL YT AS SHOWN IN FIGUREREFFIGSYST2  LET US DEFINE HOMEGA  HEJOMEGA  HZBIGLZEJOMEGATHE FIRST EQUALITY IS BY DEFINITION AND IS ACTUALLY AN ABUSE OFNOTATION  HOWEVER IT AFFORDS SOME NOTATIONAL SIMPLICITY AND IS VERYCOMMON  THEN SEE APPENDIX REFAPPDXRP THE PSD OF THE OUTPUT IS SYYOMEGA  HOMEGA2 SFFOMEGATHE SPECTRUM ESTIMATION PROBLEM IS AS FOLLOWS GIVEN A SET OFOBSERVATIONS FROM A RANDOM SIGNAL Y0Y1LDOTSYN DETERMINEESTIMATE THE PSD  IN THE PARAMETRIC APPROACH TO SPECTRUMESTIMATION WE REGARD YT AS THE OUTPUT OF A SYSTEM HZ  ITIS COMMON TO ASSUME THAT THE INPUT SIGNAL IS A ZEROMEAN WHITE SIGNALSO THAT SFFOMEGA  TEXTCONSTANT  SIGMAF2THE PARAMETERS OF HZ AND THE INPUT POWER PROVIDE THE INFORMATIONNECESSARY TO ESTIMATE THE OUTPUT SPECTRUM SYOMEGASUBSECTIONIDENTIFICATION OF THE MODESLABELSECMODAL1INDEXMODAL ANALYSIS RELATED TO SPECTRUM ESTIMATION IS THEIDENTIFICATION OF THE MODES IN A SYSTEM  WE PRESENT THE FUNDAMENTALCONCEPT USING A SECONDORDER SYSTEM WITHOUT THE COMPLICATION OF NOISEIN THE SIGNAL  ASSUME THAT A SIGNAL YT IS THE OUTPUT OF ASECONDORDER HOMOGENEOUS SYSTEMBEGINEQUATION YT2  A1 YT1  A2 YT  0LABELEQMODE1ENDEQUATIONSUBJECT TO CERTAIN INITIAL CONDITIONS  THE CHARACTERISTIC EQUATION OFTHIS SYSTEM ISBEGINEQUATION Z2  A1 Z  A2  0LABELEQMODE2ENDEQUATIONTHE MODES OF THE SYSTEM ARE DETERMINED BY THE ROOTS OF THECHARACTERISTIC EQUATION  WRITING Z2  A1 Z  A2  ZP1ZP2AND ASSUMING THAT P1 NEQ P2 THEN YT  C1P1T  C2P2T QQUAD T GEQ 0WHERE THE MODE STRENGTHS AMPLITUDES C1 AND C2 ARE DETERMINEDBY THE INITIAL CONDITIONSBASED UPON THE NOISEFREE EQUATION REFEQMODE1 WE CAN WRITE ASET OF EQUATIONS TO DETERMINE THE SYSTEM PARAMETERS A1A2 BEGINBMATRIX Y1  Y0  Y2  Y1 VDOTSENDBMATRIX  BEGINBMATRIXA1  A2 ENDBMATRIX BEGINBMATRIX Y2    Y3   VDOTS ENDBMATRIXPROVIDED THAT THE MATRIX IN THIS EQUATION HAS FULL RANK THEPARAMETERS A1 AND A2 CAN BE FOUND BY SOLVING THIS SET OFEQUATIONS FROM WHICH THE MODES CAN BE IDENTIFIED BY FINDING THE ROOTSOF REFEQMODE2  USING THIS METHOD TWO MODES CAN BE IDENTIFIEDUSING AS FEW AS FOUR MEASUREMENTS  TWO REAL SINUSOIDS WITH TWOCOMPLEX EXPONENTIAL MODES IN EACH CAN BE IDENTIFIED WITH AS FEW ASEIGHT MEASUREMENTS AND THEY CAN IN PRINCIPLE AND IN THE ABSENCE OFNOISE BE DISTINGUISHED NO MATTER HOW CLOSE IN FREQUENCY THEY AREBEGINEXAMPLE  SUPPOSE THAT YT IS KNOWN TO CONSIST OF TWO REAL SINUSOIDAL  SIGNALS YT  A COSOMEGA1 T  THETA1  B COSOMEGA2 T THETA2EACH COSINE FUNCTION CONTRIBUTES TWO MODES COSOMEGA1 T  FRACEJOMEGA1 T  EJOMEGA1 T2SO WE WILL ASSUME THAT YT IS GOVERNED BY THE FOURTHORDERDIFFERENCE EQUATION YT  A1 YT1  A2 YT2  A3 YT3  0THEN ASSUMING THAT CLEAN NOISEFREE MEASUREMENTS ARE AVAILABLE WECAN SOLVE FOR THE COEFFICIENTS OF THE DIFFERENCE EQUATION BYBEGINEQUATION BEGINBMATRIXY3  Y2  Y1  Y0 Y4  Y3  Y2  Y1 Y5  Y4  Y3  Y2 Y6  Y5  Y4  Y3 ENDBMATRIXBEGINBMATRIXA1  A2  A3  A4 ENDBMATRIX BEGINBMATRIX Y4 Y5  Y6  Y7ENDBMATRIX LABELEQMODEEX1ENDEQUATIONIF THE MEASURED OUTPUT DATA SET ISBEGINALIGNED YBF   Y0 Y1 LDOTS Y7 255433  191774  115137  033427  0451325 11354  167244  20477ENDALIGNEDTHEN SUBSTITUTION IN REFEQMODEEX1 YIELDS A1A2A3A4   3715354404371531 Z4 37153 Z3 54404Z237153Z 1WHICH HAS ROOTS AT EPM J 05QQUAD TEXTANDQQUADEPM J 02SO THE FREQUENCIES OF THE MODES ARE OMEGA1  05 AND OMEGA2 02  ONCE THE FREQUENCIES ARE KNOWN THE AMPLITUDES AND PHASES CANALSO BE DETERMINED 3COS2T  PI4  2 COS5 T  PI6ENDEXAMPLEGENERALIZATION OF THESE CONCEPTS TO A SYSTEM OF ANY ORDER IS DISCUSSEDIN SECTION REFSECMODALMAT TREATMENT OF THE MEASUREMENT NOISE ISDISCUSSED IN SECTIONS REFSECMUSIC AND REFSECESPRITSUBSECTIONCONTROL OF THE MODESINDEXCONTROLSUPPOSE WE HAVE A SYSTEM DESCRIBED BY THE DYNAMICS BEGINBMATRIXX1T1  X2T1 ENDBMATRIX  BEGINBMATRIX 05  0  0  3 ENDBMATRIXBEGINBMATRIXX1T  X2T ENDBMATRIX   BEGINBMATRIX1  1 ENDBMATRIXFTBECAUSE THE A MATRIX IS A DIAGONAL MATRIX THE STATE VARIABLEEQUATIONS ARE SAID TO BE UNCOUPLED X1T1  05X1T  FTDOES NOT DEPEND ON X2 AND X2T1   3 X2T  FTDOES NOT DEPEND UPON X1  THE QUESTION OF HOW TO PUT A GENERALSYSTEM INTO DIAGONAL INDEXDIAGONAL MATRIX FORM IS ADDRESSED INSECTION REFSECDIAGONAL  THE HOMOGENEOUS RESPONSES ZEROINPUT OFTHE MODES SEPARATELY ARE X1T  05N X10 QQUAD X2T  3N X20THE STATE VARIABLE X1T DECAYS TO ZERO AS N RIGHTARROW INFTYWHILE THE STATE VARIABLE X2T BLOWS UP  IF THIS REPRESENTED THESTATE OF A MECHANICAL SYSTEM SUCH EXPONENTIAL GROWTH WOULD PROBABLYBE UNDESIRABLE  A NATURAL QUESTION ARISES IS IT POSSIBLE TODETERMINE AN INPUT SEQUENCE FT IN CONJUNCTION WITH FEEDBACK THATCONTROLS THE SYSTEM SO THAT BOTH STATE VARIABLES REMAIN STABLE  THEMEANS OF ACCOMPLISHING THIS FALLS VERY NATURALLY INTO PLACE USING SOMETECHNIQUES FROM LINEAR ALGEBRA   SEE SECTION REFSECMOVEEIGBEGINEXERCISESITEM SYSTEM IDENTIFICATION IN THIS EXERCISE YOU WILL DEVELOP A  TECHNIQUE FOR IDENTIFICATION OF THE PARAMETERS OF A CONTINUOUSTIME  SECONDORDER SYSTEM BASED UPON FREQUENCY RESPONSE MEASUREMENTS BODE  PLOTS  ASSUME THAT THE SYSTEM TO BE IDENTIFIED HAS AN OPENLOOP  TRANSFER FUNCTION HOS  FRACBSSA  BEGINENUMERATE  ITEM SHOW THAT WITH THE SYSTEM IN A FEEDBACK CONFIGURATION AS SHOWN    IN FIGURE REFFIGBODEID1 THE TRANSFER FUNCTION CAN BE WRITTEN    AS HCS  FRACYSFS  FRAC11ABS  1BS2ITEM SHOW THAT FRAC1HCJOMEGA  AJOMEGA ANGLE PHIJOMEGAWHERE AJOMEGA  FRAC1BSQRTBOMEGA22  AOMEGA2 QQUADTEXTANDQQUAD TAN PHIJOMEGA  FRACAOMEGAB  OMEGA2THE QUANTITIES AJOMEGA AND PHIJOMEGA CORRESPOND TO THERECIPROCAL AMPLITUDE AND THE PHASE DIFFERENCE BETWEEN INPUT ANDOUTPUTITEM SHOW THAT IF AMPLITUDEPHASE MEASUREMENTS ARE MADE AT N  DIFFERENT FREQUENCIES OMEGA1 OMEGA2 LDOTS OMEGAN THEN  THE UNKNOWN PARAMETERS A AND B CAN BE ESTIMATED BY SOLVING THE  OVERDETERMINED SET OF EQUATIONS BEGINBMATRIX AJOMEGA1  OMEGA1 SQRT1 1TAN2 PHIJOMEGA1 TAN PHIJOMEGA1  OMEGA1 AJOMEGA2  OMEGA2 SQRT1 1TAN2 PHIJOMEGA2 TAN PHIJOMEGA2  OMEGA2 VDOTS AJOMEGAN  OMEGAN SQRT1 1TAN2 PHIJOMEGAN TAN PHIJOMEGAN  OMEGAN ENDBMATRIXBEGINBMATRIX B  A ENDBMATRIX BEGINBMATRIX 0  OMEGA12 TANPHIJOMEGA1 0  OMEGA22 TANPHIJOMEGA2 VDOTS 0  OMEGAN2 TANPHIJOMEGAN ENDBMATRIX  ENDENUMERATE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRFEEDBACK1      CAPTIONSIMPLE FEEDBACK CONFIGURATION      LABELFIGBODEID1    ENDCENTER  ENDFIGUREITEM VERIFY REFEQESD1  LABELEXESD1ITEM SHOW THAT  SUMNINFTYINFTY YN2  FRAC12PI INTPIPISOMEGA DOMEGAHINT RECALL THE INVERSE FOURIER TRANSFORM INDEXFOURIER TRANSFORM YT FRAC12PI INTPIPI YOMEGA EJ OMEGA TDOMEGAITEM SHOW THAT FOR A STOCHASTIC SIGNAL SOMEGA  LIMNRIGHTARROW INFTY ELEFT FRAC1N  LEFTSUMN1N YN EJOMEGA NRIGHT2 RIGHTHINT  SHOW AND USE THE FACT THAT SUMN1N SUMM1N FNM  SUMLN1N1 NLFLITEM MODAL ANALYSIS THE FOLLOWING DATA IS MEASURED FROM A  THIRDORDER SYSTEM Y   0320002500010000022200006000120000500001BEGINENUMERATEITEM DETERMINE THE MODES IN THE SYSTEM AND PLOT THEM IN THE COMPLEX PLANEITEM THE DATA CAN BE WRITTEN AS YT  C1P1T  C2P2T  C3P3T QQUAD T GEQ 0DETERMINE THE CONSTANTS C1 C2 AND C3ITEM TO EXPLORE THE EFFECT OF NOISE ON THE SYSTEM ADD RANDOM  GAUSSIAN NOISE TO EACH DATA POINT WITH VARIANCE SIGMA2  001  THEN FIND THE MODES OF THE NOISY DATA  REPEAT SEVERAL TIMES WITH  DIFFERENT NOISE AND COMMENT ON HOW THE MODAL ESTIMATES MOVEENDENUMERATEITEM MODAL ANALYSIS  IF YT HAS TWO REAL SINUSOIDS YT  A COSOMEGA1 T  THETA1  B COSOMEGA2 T THETA2AND THE FREQUENCIES ARE KNOWN DETERMINE A MEANS OF COMPUTING THEAMPLITUDES AND PHASESENDEXERCISESSECTIONADAPTIVE FILTERINGLABELSECADFILTINDEXADAPTIVE FILTERAN ADAPTIVE FILTER IS A FILTER USUALLY WITH AN FIR IMPULSERESPONSE IN WHICH THE COEFFICIENTS ARE OBTAINED BY ATTEMPTING TOFORCE THE OUTPUT OF THE FILTER YT TO MATCH SOME DESIRED INPUTSIGNAL DT    SCHEMATICALLY THE FILTER IS SHOWN IN FIGUREREFFIGADFILT1  THE ERROR SIGNAL  ET  DT  YTIS USED IN SPECIALIZED ALGORITHMS THE ADAPTATION RULE TO ADJUST THECOEFFICIENTS OF THE ADAPTIVE FILTER  A VARIETY OF ADAPTATION RULESARE EMPLOYED IN PARTICULAR WE WILL STUDY THE RECURSIVE LEASTSQUARESRLS ALGORITHM PRESENTED IN SECTION REFSECRLS AND THE LEAST MEANSQUARES LMS ALGORITHM PRESENTED IN SECTION REFSECLMSBEGINFIGUREHTBPBEGINCENTERINPUTPICTUREDIRADFILT1ENDCENTERCAPTIONREPRESENTATION OF AN ADAPTIVE FILTER  LABELFIGADFILT1ENDFIGUREADAPTIVE FILTERS ARE EMPLOYED IN A VARIETY OF CONFIGURATIONS SOME OFWHICH ARE HIGHLIGHTED IN THIS SECTIONSUBSECTIONSYSTEM IDENTIFICATIONINDEXSYSTEM IDENTIFICATIONAN ADAPTIVE FILTER CAN ESTIMATE THE THE TRANSFER FUNCTION OF ANUNKNOWN PLANT USING THE CONFIGURATION SHOWN IN FIGUREREFFIGADFILSYSID  THE ADAPTIVE FILTER AND THE PLANT ARE BOTHDRIVEN BY THE SAME INPUT SIGNAL AND THE DESIRED SIGNAL DT IS THEPLANT OUTPUT  THE ADAPTIVE FILTER WILL CONVERGE TO A BESTREPRESENTATION OF THE UNKNOWN SYSTEM  IF THE SYSTEM IS AN IIR SYSTEMAND THE ADAPTIVE FILTER IS AN FIR SYSTEM OR IF THE ORDER OF THEADAPTIVE FILTER IS LESS THAN THE ORDER OF THE SYSTEM THEN THEADAPTIVE FILTER CAN BE AT BEST AN APPROXIMATION OF THE TRUE SYSTEMRESPONSEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADFILTSYSID    CAPTIONIDENTIFICATION OF AN UNKNOWN PLANT    LABELFIGADFILSYSID  ENDCENTERENDFIGURESUBSECTIONINVERSE SYSTEM IDENTIFICATIONINDEXINVERSE SYSTEM IDENTIFICATIONWHEN THE ADAPTIVE FILTER IS CONFIGURED AS SHOWN IN FIGUREREFFIGADFILTINVSYS THEN IT WILL CONVERGE WHEN THE OUTPUT OF THEADAPTIVE FILTER MATCHES THE DELAYED INPUT OF THE INVERSE SYSTEM ASCLOSELY AS POSSIBLE  IDEALLY THE ADAPTIVE FILTER WILL CONVERGE TOTHE INVERSE OF THE PLANT SO THAT THE CASCADE OF THE PLANT AND THEADAPTIVE FILTER IS SIMPLY A DELAY  THIS CONFIGURATION IS EMPLOYED INSOME MODEMS TO REDUCE THE EFFECT OF THE CHANNEL ON THE TRANSMITTEDSIGNAL  THE SIGNAL REPRESENTING A SEQUENCE OF INPUT BITS FTPASSES THROUGH A CHANNEL WITH AN UNKNOWN TRANSFER FUNCTION HZ  ATTHE RECEIVER THE SIGNAL IS PROCESSED BY AN ADAPTED INVERSE SYSTEMBEFORE DETECTING THE BITSAN EXAMPLE OF THE OPERATION IN THISCONFIGURATION IS PROVIDED IN SECTION REFSECRLSBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADFILTINVSYS    CAPTIONADAPTING TO THE INVERSE OF AN UNKNOWN PLANT    LABELFIGADFILTINVSYS  ENDCENTERENDFIGURESUBSECTIONADAPTIVE PREDICTORSINDEXLINEAR PREDICTION IN THE CONFIGURATION SHOWN IN FIGUREREFFIGADFILPREDICTOR THE INPUT TO THE ADAPTIVE FILTER IS ADELAYED VERSION OF THE DESIRED SIGNAL  IN THIS CASE THE ADAPTIVEFILTER CONVERGES IN SUCH A WAY AS TO PROVIDE A PREDICTOR OF THE INPUTSIGNAL IF PREDICTION IS POSSIBLE  IN THIS MODE IT CAN BE USED FORALL THE APPLICATIONS MENTIONED PREVIOUSLY FOR LINEAR PREDICTORSINCLUDING DATA COMPRESSION PATTERN RECOGNITION OR SPECTRUMESTIMATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADFILTPRED    CAPTIONAN ADAPTIVE PREDICTOR    LABELFIGADFILPREDICTOR  ENDCENTERENDFIGURESUBSECTIONINTERFERENCE CANCELLATIONINDEXINTERFERENCE CANCELLATIONIN THE CONTEXT OF INTERFERENCE CANCELLATION THE SIGNAL DT ISCOMMONLY REFERRED TO AS THE PRIMARY SIGNAL WHILE THE FILTER INPUTIS REFERRED TO AS THE SECONDARY SIGNAL  THE PRIMARY DT ISMODELED AS THE SUM OF A SIGNAL OF INTEREST XT PLUS  NOISE DT  XT  WTTHE SECONDARY INPUT CONSISTS OF A NOISE SIGNAL FT  NTSEE FIGURE REFFIGADCANCELBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADCANCEL    CAPTIONCONFIGURATION FOR INTERFERENCE CANCELLATION    LABELFIGADCANCEL  ENDCENTERENDFIGUREAS AN EXAMPLE SUPPOSE THAT A BACKGROUND ACOUSTIC NOISE SOURCE SAYTHE HUM OF A FAN WT SUPERIMPOSED ON A DESIRED AUDIO SIGNALXT WHICH IS RECORDED USING A MICROPHONE TO FORM THE PRIMARYINPUT  A SECOND MICROPHONE PLACED FAR FROM THE DESIRED SIGNAL RECORDSTHE NOISE NT BUT NOT THE DESIRED SIGNAL  THERE IS A DIFFERENTACOUSTIC TRANSFER FUNCTION BETWEEN THE SOURCE AND EACH OF THE TWOMICROPHONES HENCE NT IS NOT THE SAME AS WT  THE ADAPTIVEFILTER IS DRIVEN TO MINIMIZE THE ERROR WHICH ADAPTS TO ACCOMMODATETHIS DIFFERENCE IN TRANSFER FUNCTION FROM THE NOISE SOURCE  THUS THERESULTING DIFFERENCE SIGNAL ET WILL HAVE INSOFAR AS POSSIBLETHE NOISE FROM THE REFERENCE SIGNAL SUBTRACTED FROM THE NOISE FROM THEPRIMARY SIGNALTHE INTERFERENCE CANCELLATION CONFIGURATION HAS BEEN USED IN SEVERALAPPLICATIONS SUCH AS NOISE CANCELLATION ECHO CANCELLATION ANDADAPTIVE BEAMFORMING IN ARRAY PROCESSINGSECTIONGAUSSIAN RANDOM VARIABLES AND RANDOM PROCESSESLABELSECMULTGAUSSINDEXGAUSSIAN RANDOM VARIABLE INDEXRANDOM VARIABLEGAUSSIANINDEXNORMAL RANDOM VARIABLESEEGAUSSIAN RANDOM VARIABLE WE BEGINBY REVIEWING THE BASIC PROPERTIES OF SINGLE GAUSSIAN RANDOM VARIABLESSEE BOX REFBOXPROBNOT FOR NOTATIONAL CONVENTIONS  LET W BE AGAUSSIAN RANDOM VARIABLE WITH MEAN MU AND VARIANCE SIGMA2NOTATIONALLY WE  WRITE  W SIM NCMUSIGMA2THE SCALAR GAUSSIAN PROBABILITY DENSITY FUNCTION PDF SHOULD BEFAMILIARBOXED FWW  FRAC1SIGMASQRT2PI EW     MU22SIGMA2WHERE MU IS THE MEAN AND SIGMA2 IS THE VARIANCE OF THEDISTRIBUTION  THAT IS MU  EW  INTINFTYINFTY W FWW DW FRAC1SQRT2PI SIGMA INTINFTYINFTY W EW   MU22SIGMA2 DWAND SIGMA2  EWMU2  EW2  MU2  FRAC1SQRT2PI  SIGMA INTINFTYINFTY W2 EW  MU22SIGMA2 DW  MU2FIGURE REFFIGGAUSS1 ILLUSTRATES A GAUSSIAN PDF WITH MU0 ANDSIGMA2 1BEGINFIGURE PLOTGAUSSMCENTERLINEEPSFIGFILEPICTUREDIRPLOTGAUSSEPSCAPTIONTHE GAUSSIAN DENSITYLABELFIGGAUSS1ENDFIGUREBEGINTEXTBOX09TEXTWIDTHNOTATION FOR RANDOM VARIABLES AND VECTORSLABELBOXPROBNOTSCALAR RANDOM VARIABLES ARE REPRESENTED USING CAPITAL LETTERS WHILE APARTICULAR OUTCOME VALUE FOR A RANDOM VARIABLE IS INDICATED IN LOWERCASE USUALLY THE SAME LETTER THUS X IS A RANDOM VARIABLE AND XMAY BE AN OUTCOME OF THE RANDOM VARIABLE  INDEXFONTSCAPITALINDEXCAPITAL LETTERSSEEFONTS RANDOM VECTORS ARE USUALLYPRESENTED AS BOLD CAPITAL LETTERS  WHERE THE NOTATION OF THELITERATURE COMMONLY EMPLOYS LOWER CASE WE FOLLOW SUITINDEXFONTSBOLD CAPITALINDEXPROBABILITY DENSITY FUNCTION PDF INDEXPROBABILITY MASS  FUNCTION PMF BOXINDENT A PROBABILITY DENSITY FUNCTION PDF ORPROBABILITY MASS FUNCTION PMF FOR A RANDOM VARIABLE X IS WRITTENAS FXX  HOWEVER IT WILL BE COMMON THROUGHOUT THE TEXT TOSUPPRESS THE SUBSCRIPT NOTATION LETTING THE ARGUMENT OF THE FUNCTIONPROVIDE THE INDICATION OF THE RANDOM VARIABLE  THUS WE WILLFREQUENTLY WRITE FX TO MEAN FXXENDTEXTBOXASSOCIATED WITH THE GAUSSIAN PDF ARE THE FOLLOWING USEFUL INTEGRALSTRUE FOR ALL VALUES OF MU AND SIGMA NEQ 0BEGINEQUATIONBOXEDFRAC1SIGMASQRT2PI INTINFTYINFTY  EXMU22SIGMA2  DX  1LABELEQGAUSSINT1ENDEQUATIONBEGINEQUATIONBOXEDFRAC1SIGMA SQRT2PI INTINFTYINFTY X  EXMU22SIGMA2  DX  MULABELEQGAUSSINT2ENDEQUATIONBEGINEQUATIONBOXEDFRAC1SIGMA SQRT2PI INTINFTYINFTY X2  EXMU22SIGMA2  DX  SIGMA2  MU2LABELEQGAUSSINT3ENDEQUATIONMEASURED SIGNALS ARE COMMONLY CORRUPTED BY NOISE  IF YBFTREPRESENTS A VECTOR SYSTEM OUTPUT THE MEASURED VALUE IS OFTEN MODELEDAS ZBFT  YBFT  WBFTWHERE WBFT IS A VECTOR OF NOISE SAMPLES  WBFT  BEGINBMATRIX W1T  W2T  VDOTS  WKTENDBMATRIXTHIS IS THE SIGNAL PLUS NOISE MODEL INDEXSIGNAL PLUS NOISEIN THE ABSENCE OF SPECIFIC REASONS TO THE CONTRARY IT IS COMMON TOASSUME THAT ADDITIVE NOISE SIGNALS ARE DISTRIBUTED WITH A EM  GAUSSIAN OR NORMAL DISTRIBUTION  QUANTIZATION NOISE IS ANEXCEPTION TO THIS ASSUMPTION IT IS USUALLY MODELED AS A UNIFORMRANDOM VARIABLE  THERE ARE REASONS FOR ASSUMING THAT RANDOMVARIABLES AND RANDOM PROCESSES ARE GAUSSIAN  FIRST GAUSSIAN NOISEOCCURS PHYSICALLY  FOR EXAMPLE THE THERMAL NOISE AT THE FRONT END OFA RADIO RECEIVER IS OFTEN GAUSSIAN  SECOND GAUSSIAN NOISE SIGNALSHAVE A VARIETY OF USEFUL PROPERTIES WHICH SIMPLIFY SEVERAL THEORETICALDEVELOPMENTS  SOME OF THESE PROPERTIES ARE AS FOLLOWSINDEXGAUSSIAN RANDOM VARIABLEATTRIBUTESBEGINENUMERATEITEM BY THE CENTRAL LIMIT THEOREM THE DISTRIBUTION OF SUMS OF  SEVERAL RANDOM VARIABLES TENDS TOWARD A GAUSSIAN DISTRIBUTION  MORE  PRECISELY IF X1 X2 LDOTS XN ARE INDEPENDENT RANDOM  VARIABLES WITH MEANS MU1 MU2 LDOTS MUN AND VARIANCES  SIGMA12 SIGMA22 LDOTS SIGMAN2 RESPECTIVELY THEN Y  SUMI1N FRACXI  MUISIGMAIHAS A DISTRIBUTION WHICH APPROACHES A GAUSSIAN DISTRIBUTION WITH MEAN0 AND VARIANCE 1 AS N BECOMES LARGE ENOUGH  IN THE LIMIT AS NRIGHTARROW INFTY THEN Y SIM NC01  THE CENTRAL LIMIT THEOREMACCOUNTS IN LARGE MEASURE FOR THE OCCURRENCE OF GAUSSIAN NOISE INPRACTICE THE MEASURED NOISE IS ACTUALLY THE SUM OF MANY SMALLINDEPENDENT EFFECTSBEGINEXAMPLE  AN APPRECIATION OF THE CENTRAL LIMIT THEOREM CAN BE GAINED BY  LOOKING AT THE SUM OF ONLY THREE VARIABLES  LET X1 X2 AND  X3 BE INDEPENDENT RANDOM VARIABLES UNIFORMLY DISTRIBUTED FROM  12 TO 12  NOTATIONALLY WE WRITE XI SIM UC1212  THE PDF FOR THIS UNIFORM RANDOM VARIABLE IS SHOWN IN FIGURE  REFFIGPDF1A  LET Z  X1  X2  KEEP IN MIND THAT THE PDF  OF THE SUM OF INDEPENDENT RANDOM VARIABLES IS THE CONVOLUTION OF THE  PDFS INDEXCONVOLUTION THE PDF OF Z IS THUS THE HAT SHAPED  FUNCTION SHOWN IN FIGURE REFFIGPDF1B THE CONVOLUTION OF TWO  FLAT PULSES  LET Y  ZX3  X1  X2  X3  THE PDF OF Y  OBTAINED AGAIN BY CONVOLUTION IS SHOWN IN FIGURE REFFIGPDF1C  THIS IS A PIECEWISE QUADRATIC FUNCTION BUT OBSERVE HOW IT IS  ALREADY BEGINNING TO LOOK LIKE THE GAUSSIAN DENSITY IN FIGURE  REFFIGGAUSS1BEGINFIGUREHTBP  CENTERINGSUBFIGUREFXXEPSFIGFILEPICTUREDIRUNIF1EPSSUBFIGUREFZZEPSFIGFILEPICTUREDIRUNIF2EPSSUBFIGUREFYYEPSFIGFILEPICTUREDIRUNIF3EPS PLOTGAUSS2M  CAPTIONDEMONSTRATION OF THE CENTRAL LIMIT THEOREM  LABELFIGPDF1ENDFIGUREENDEXAMPLEITEM A GAUSSIAN RANDOM VARIABLE W IS ENTIRELY DETERMINED BY ITS MEAN  AND ITS VARIANCE  A GAUSSIAN RANDOM PROCESS WT IS DETERMINED BY ITS  MEAN MWT  EWTAND AUTOCORRELATIONBEGINEQUATION RWTS  EWTWBARSLABELEQGAUSSCORRCH1ENDEQUATIONA GAUSSIAN RANDOM PROCESS WITH CONSTANT MEAN SUCH THAT RWTS RWST THAT IS WITH THE AUTOCORRELATION DEPENDING UPON THE TIMEDIFFERENCE IN SAMPLE POINTS IS STATIONARYITEM LINEAR OPERATIONS ON GAUSSIAN RANDOM VARIABLES PRODUCE GAUSSIAN  RANDOM VARIABLES  THAT IS IF X AND Y ARE JOINTLY GAUSSIAN  THEN Z  AX  BYIS ALSO GAUSSIAN FOR ANY CONSTANTS A AND B  IN PARTICULAR THESUM OF GAUSSIANS IS GAUSSIAN  THIS FOLLOWS SINCE THE CONVOLUTION OFGAUSSIANS IS GAUSSIAN  FURTHERMORE IF A GAUSSIAN RANDOM PROCESS IS INPUT TO A LINEAR SYSTEMTHEN THE OUTPUT IS ALSO A GAUSSIAN RANDOM PROCESS  ALL THAT MUST BEDETERMINED IS THE MEAN AND AUTOCORRELATION OF THE OUTPUT SIGNAL ANDIT IS FULLY CHARACTERIZED ITEM MAXIMUM LIKELIHOOD DETECTION OR ESTIMATION INVOLVING GAUSSIAN  RANDOM VARIABLES CORRESPONDS TO A EUCLIDEAN DISTANCE METRIC  THIS  IS GENERALLY GEOMETRICALLY PALATABLE AND ANALYTICALLY TRACTABLEITEM WIDESENSE STATIONARY WSS GAUSSIAN RANDOM PROCESSES ARE ALSO  STRICTSENSE STATIONARY SSS SEE APPENDIX REFAPPDXRP  INDEXWIDESENSE STATIONARYITEM UNCORRELATED GAUSSIAN RANDOM VARIABLES ARE ALSO INDEPENDENTITEM A GAUSSIAN CONDITIONED UPON A GAUSSIAN IS GAUSSIAN  INDEXGAUSSIAN RANDOM VARIABLECONDITIONAL DENSITYENDENUMERATEJUSTIFICATIONS FOR MANY OF THESE PROPERTIES ARE PROVIDED THROUGHOUTTHIS BOOK AS THEY ARISEFOR A GAUSSIAN RANDOM VECTOR WBF OF DIMENSION K WITH MEAN MUBFAND COVARIANCE MATRIX R WE WRITE WBF SIM NCMUBF R  THEPDF ISBEGINEQUATIONBOXED FWBFWBF  FRAC12PIK2 R12 EXPFRAC12WBF  MUBFT R1WBF  MUBFLABELEQMULTGAUSSENDEQUATIONWHERE MUBF IS THE MEAN MUBF  EWBF  BEGINBMATRIX EW1  EW2  VDOTS   EWK ENDBMATRIXAND R IS THE MATSIZEKK COVARIANCE MATRIX R  EWBFMUBFWBF  MUBFT  EWBFWBFT  MUBF MUBFTINDEXBAR CDOT INDEX  CDOT THE NOTATION R IN REFEQMULTGAUSS INDICATES THE ABSOLUTEVALUE OF THE DETERMINANT OF THE MATRIX R SEE SECTIONREFSECDETERM  IN OTHER CONTEXTS THE NOTATION R WILLINDICATE THE DETERMINANT BUT THE ABSOLUTE VALUE IS NEEDED IN THISCASE SINCE A DENSITY FUNCTION IS ALWAYS NONNEGATIVEMANY OF THE SIGNIFICANT CONCEPTS ASSOCIATED WITH GAUSSIAN RANDOMVECTORS CAN BE OBTAINED BY EXAMINATION OF TWODIMENSIONAL VECTORSWHEN WBF  W1W2TBEGINEQUATION R  BEGINBMATRIX SIGMA12  SIGMA12  SIGMA12   SIGMA22 ENDBMATRIXLABELEQR22ENDEQUATIONWHERE  SIGMA12  EW12  MU12 QQUAD SIGMA22  EW22  MU22 AND SIGMA12  EW1 W2  MU1 MU2THE EM  CORRELATION COEFFICIENT IS DEFINED ASBEGINEQUATION RHO  FRACEW1 W2  MU1 MU2 SIGMA1 SIGMA2LABELEQCORRCOEFFENDEQUATIONUSING THE CAUCHYSCHWARZ INEQUALITY WHICH IS INTRODUCED IN SECTIONINDEXCAUCHYSCHWARZ INEQUALITY REFSECCS IT CAN BE SHOWN THAT 1 LEQ RHO LEQ 1THE CORRELATION COEFFICIENT PROVIDES INFORMATION ABOUT HOW W1VARIES WITH W2  IF RHO  1 THEN W1  W2 AND W1 TELLSEVERYTHING THERE IS TO KNOW ABOUT W2 AND VICE VERSA  IF RHO 1 THEN W1  W2  IF RHO  0 THEN THE VARIABLES ARE SAID TOBE EM UNCORRELATED INDEXUNCORRELATED W1 DOES NOT PROVIDE ANYINFORMATION ABOUT W2  MORE GENERALLY FOR A KDIMENSIONAL RANDOMVECTOR WBF IF THE CORRELATION MATRIX R IS DIAGONALINDEXDIAGONAL MATRIX THE COMPONENTS OF WBF ARE UNCORRELATEDWE CAN WRITE THE INVERSE OF THE COVARIANCE MATRIX REFEQR22 INTERMS OF THE CORRELATION COEFFICIENT AND VARIANCES ASBEGINEQUATIONR1  FRAC11RHO2 BEGINBMATRIX FRAC1SIGMA12   FRAC RHOSIGMA1 SIGMA2 EXMATSP  FRAC RHOSIGMA1 SIGMA2  FRAC1SIGMA22 ENDBMATRIXLABELEQINVCOVARENDEQUATIONTHE JOINT PDF OF W1 AND W2 CAN NOW BE WRITTEN ASBEGINEQUATIONBEGINSPLITFW1W2   FRAC12PI SIGMA1 SIGMA2 SQRT1RHO2   EXPLEFTFRAC121RHO2LEFT FRACW1  MU12      SIGMA12   RIGHT RIGHT  QQUAD LEFT LEFT FRACW2  MU22SIGMA22  FRAC2RHOW1       MU1W2  MU2SIGMA1 SIGMA2RIGHT RIGHTLABELEQ2GAUSSENDSPLITENDEQUATIONA SURFACECURVE PLOT OF THIS FUNCTION IS SHOWN IN FIGUREREFFIG2GAUSPLOT FOR MUX  MUY  0 SIGMAX2 SIGMAY21 FOR TWO VALUES OF RHOBEGINFIGUREHTBPCENTERINGSUBFIGURERHO09EPSFIGFILEPICTUREDIRGAUSS21EPSWIDTH045TEXTWIDTHSUBFIGURERHO0EPSFIGFILEPICTUREDIRGAUSS22EPSWIDTH045TEXTWIDTHPLOTGAUSS3MCAPTIONPLOT OF TWODIMENSIONAL GAUSSIAN DISTRIBUTIONLABELFIG2GAUSPLOTENDFIGUREIN REFEQ2GAUSS IF RHO0 THEN FW1W2  FRAC12PI SIGMA1 SIGMA2EXPLEFTFRAC12  LEFT FRACW1  MU12SIGMA12  FRACW2       MU22SIGMA22RIGHT RIGHT  FW1FW2SUBSTANTIATING THE CLAIM MADE PREVIOUSLY THAT UNCORRELATED GAUSSIANRANDOM VARIABLES ARE INDEPENDENTSUBSECTIONCONDITIONAL GAUSSIAN DENSITIESLABELSECCONDESTINDEXGAUSSIAN RANDOM VARIABLECONDITIONAL DENSITY CONDITIONALPROBABILITIES CONSTITUTE THE CORE OF MANY DETECTION AND ESTIMATIONALGORITHMS  IN THIS SECTION WE PRESENT A SIMPLE EXAMPLE OFCONDITIONING AS A FORERUNNER TO THE MORE COMPLETE DEVELOPMENT OFSTATISTICAL DECISION MAKING IN PART REFPARTDETESTSUPPOSE THAT X AND Y ARE JOINTLY GAUSSIAN RANDOM VARIABLES XSIM NCMUX SIGMAX2 Y SIM NCMUY SIGMAY2 WITHCORRELATION COEFFICIENT RHO  WE WANT TO EM ESTIMATE A VALUE FORX WHICH WE WILL DENOTE AS XHAT  IN THE ABSENCE OF ANY INDEXESTIMATIONMEASUREMENTS A REASONABLE VALUE FOR XHAT IS SIMPLY THE MEAN OFX SO XHAT  MUXINDEXCONDITIONAL PROBABILITY SUCH AN ESTIMATE  OBTAINABLEWITHOUT THE BENEFIT OF ANY MEASUREMENTS  IS A EM PRIOR OR EM A  PRIORI INDEXPRIOR ESTIMATE ESTIMATE AND THE DENSITY FXX ISKNOWN AS THE EM A PRIORI DENSITY FOR X  WHEN A MEASUREMENT OFY IS AVAILABLE SAY YY THEN THIS CAN BE USED TO MODIFY OUR PRIORESTIMATE OF X SINCE X AND Y ARE CORRELATED  ONE APPROACH TOTHIS IS TO FORM THE CONDITIONAL PDF FXYXY THE DENSITY OF XGIVEN THAT YY IS KNOWN AND DETERMINE OUR ESTIMATE XHAT BY THEMEAN OF THIS NEW DENSITY  THE CONDITIONAL DENSITY IS DEFINED ASINDEXCONDITIONAL PROBABILITY FXYXY  FXY  FRACFXYFYFROM REFEQ2GAUSS WITH X  W1 AND YW2 WE OBTAININDEXCONDITIONAL PROBABILITYGAUSSIANBEGINEQUATIONBEGINSPLITFXY  FRAC FRAC12PI SIGMAX SIGMAY SQRT1RHO2  EXPLEFT     FRAC121RHO2LEFTFRACXMUX2SIGMAX2       FRACYMUY2SIGMAY2  FRAC2RHOSIGMAX SIGMAY      XMUXYMUYRIGHTRIGHTFRAC1SQRT2PI SIGMAY  EXPFRAC12SIGMAY2YMUY2   FRAC1SQRT2PI1RHO2 SIGMAX EXPLEFTFRAC12 SIGMAX2  SQRT1RHO2 XMUX FRACSIGMAXSIGMAYRHOYMUY2RIGHTENDSPLITLABELEQFXYENDEQUATIONTHE ALGEBRA HERE REQUIRES COMPLETING THE SQUARE AS DESCRIBED ININDEXCOMPLETING THE SQUAREAPPENDIX REFAPPDXCTS  FROM THE FORM OF THE PDF WE RECOGNIZE THATFXY IS GAUSSIAN WITH MEANBEGINEQUATION EXY  MUX  FRACSIGMAXSIGMAYRHOYMUYLABELEQCONDMEANGAUSS1ENDEQUATIONAND VARIANCEBEGINEQUATION VARXY  SIGMAX2 SQRT1RHO2LABELEQCONDVARGAUSS1ENDEQUATIONIF X AND Y ARECORRELATED THAT IS RHO NEQ 0 THEN KNOWING Y SHOULD TELL USSOMETHING ABOUT X  BASED ON THE INFORMATION AVAILABLE ABOUT Y AREASONABLE ESTIMATE OF X IS THE CONDITIONAL MEANBEGINEQUATION XHAT  MUX  FRACSIGMAXSIGMAYRHOYMUYLABELEQCONDMEAN0ENDEQUATIONTHE VARIANCE OF THIS ESTIMATE IS THE CONDITIONAL VARIANCE OFREFEQCONDVARGAUSS1  WE CAN MAKE A MEANINGFUL INTERPRETATION OFTHE ESTIMATE REFEQCONDMEAN0  IF THERE IS NO CORRELATION THECONDITIONAL MEAN IS THE SAME AS THE PRIOR MEAN  IF RHO IS SMALLWE MAKE ONLY A SMALL MODIFICATION TO THE PRIOR MEAN  IF SIGMAY ISLARGE THEN THE CORRECTION TO THE PRIOR MEAN IS SMALL AS IT SHOULD BEIF WE HAVE LARGE UNCERTAINTY ABOUT THE OUTCOME Y  WE ALSO OBSERVETHAT INCORPORATING INFORMATION ABOUT Y REDUCES THE VARIANCE IN X SIGMAX2 SQRT1RHO2 LEQ SIGMAX2SINCE RHO LEQ 1THIS CONDITIONAL DENSITY WITH ONLY TWO VARIABLES IS EXTENDED INSECTION REFSECINVPART TO GENERAL GAUSSIAN VECTORS CONDITIONED ONGAUSSIAN VECTORSTHIS EXAMPLE INTRODUCES AN IMPORTANT PART OF ESTIMATION THEORY  ANOBSERVED OR MEASURED VARIABLE SUCH AS Y IN THE FOREGOING CAN BEUSED TO MODIFY OUR UNDERSTANDING OF VARIABLES THAT WE HAVE NOTMEASURED OR CANNOT MEASURE  A POWERFUL EXTENSION OF THIS SIMPLEEXAMPLE IS THE KALMAN FILTER IN WHICH THE STATE OF A SYSTEM IN RANDOMNOISE SUCH AS IN REFEQSTATEGEN1 IS ESTIMATED BASED UPONOBSERVATIONS THAT ARE ALSO IN NOISE  IN THE KALMAN FILTER THEDENSITY OF THE STATE VARIABLE FXBFT IS MODIFIED BY THEOBSERVATION YBFT TAKING INTO ACCOUNT THE DYNAMICS OF THE SYSTEMAND THE MECHANISM FOR OBSERVATION  THE KALMAN FILTER IS DISCUSSED INCHAPTER REFCHAPKALMAN INDEXKALMAN FILTERSEVERAL OTHER EXTENSIONS AND ISSUES NOW ARISE AMONG THEMBEGINITEMIZEITEM GIVEN A SEQUENCE OF DATA FROM SOME SOURCE WHICH IS ASSUMED TO BE  DRAWN ACCORDING TO A GAUSSIAN DISTRIBUTION HOW CAN THE PARAMETERS  OF THE GAUSSIAN DISTRIBUTION BE ESTIMATED  HOW CAN THE QUALITY OF  THE ESTIMATES BE ASSESSED  THESE QUESTIONS ARE ANSWERED IN PART BY  EM ESTIMATION THEORY  AN EARLY ANSWER IS EXPLORED IN EXERCISE  REFEXGAUSSESTITEM IF A SIGNAL IS CHOSEN AT RANDOM FROM AMONG A DISCRETE SET OF  SIGNALS AND THEN OBSERVED IN ADDITIVE NOISE HOW CAN THE CHOSEN  SIGNAL BE DISCRIMINATED  THIS IS THE EM DETECTION PROBLEM WHICH  LIES AT THE HEART OF DIGITAL COMMUNICATIONITEM GIVEN CORRELATED RANDOM VECTORS XBF AND YBF HOW CAN THE  CONDITIONAL DENSITY FXBFYBF BE COMPUTED HOW MAY THIS BE  APPLIEDITEM HOW CAN GAUSSIAN RANDOM VARIABLES OF GIVEN PARAMETERS BE  GENERATED AND USED IN SIMULATION FOR TESTING OF SIGNAL PROCESSING  ALGORITHMS  AN ANSWER FOR SCALAR GAUSSIAN RVS IS FOUND IN  EXERCISE REFEXGENGAUSSENDITEMIZEBEGINEXERCISES  ITEM SHOW THAT R1 FROM REFEQINVCOVAR IS CORRECTITEM SHOW THAT REFEQ2GAUSS FOLLOWS FROM REFEQMULTGAUSS AND  REFEQINVCOVARITEM SUPPOSE THAT XSIM NCMUXSIGMAX2 AND N SIM  NC0SIGMAN2 ARE INDEPENDENTLY DISTRIBUTED GAUSSIAN RVS  LET Y  XNBEGINENUMERATEITEM DETERMINE THE PARAMETERS OF THE DISTRIBUTION OF YITEM IF YY IS MEASURED WE CAN ESTIMATE X BY COMPUTING THE  CONDITIONAL DENSITY FXY  DETERMINE THE MEAN AND VARIANCE OF  THIS CONDITIONAL DENSITY  INTERPRET THESE RESULTS IN TERMS OF  GETTING INFORMATION ABOUT X I SIGMAN2 GG SIGMAX2 AND  II SIGMAN2 LL SIGMAX2ENDENUMERATEITEM SUPPOSE THAT X SIM NCMUX SIGMAX2  AND Y  SIMNCMUY  SIGMAY2 ARE JOINTLY DISTRIBUTED GAUSSIAN RVS  WITH CORRELATION RHO  DETERMINE THE PARAMETERS OF THE  DISTRIBUTION OF Z  A X  BYITEM IF X SIM NC01 SHOW THAT Y  SIGMA X  MUIS DISTRIBUTED AS Y NCSIGMA2MU LABELEXGENGAUSSITEM IF X SIM NCSIGMA2MU DETERMINE EX THE EXPECTED  VALUE OF THE ABSOLUTE VALUE OF XITEM LABELEXGAUSSEST LET X1 X2 LDOTS XN BE N  INDEPENDENT OBSERVATIONS OF A GAUSSIAN RANDOM VARIABLE X WITH  UNKNOWN MEAN AND VARIANCE  WE DESIRE TO ESTIMATE THE MEAN AND  VARIANCE OF X  THE JOINT DENSITY OF N INDEPENDENT GAUSSIAN  RVS CONDITIONED ON KNOWING THE MEAN MU AND THE VARIANCE  SIGMA2 IS FX1X2LDOTSXNMU SIGMA2 FRAC12PIN2SIGMANEXPFRAC12 SIGMA2SUMI1N XI  MU2BEGINENUMERATE  ITEM DETERMINE A EM MAXIMUM LIKELIHOOD ESTIMATE OF MU BY    INDEXMAXIMUM LIKELIHOOD ESTIMATION    MAXIMIZING THIS JOINT DENSITY WITH RESPECT TO MU IE TAKE THE    DERIVATIVE WITH RESPECT TO MU  CALL THE ESTIMATE OF THE MEAN    YOU OBTAIN MUHAT  ITEM SINCE MUHAT IS A FUNCTION OF RANDOM VARIABLES IT IS ITSELF    A RANDOM VARIABLE  DETERMINE THE MEAN EXPECTED VALUE OF    MUHAT  AN ESTIMATE WHOSE EXPECTED VALUE IS EQUAL TO THE VALUE    IT IS ESTIMATED IS SAID TO BE EM UNBIASED INDEXUNBIASED  ITEM DETERMINE THE VARIANCE OF MUHAT    ITEM DETERMINE AN ESTIMATE FOR SIGMA2  ENDENUMERATE  IT IS NATURAL TO ASK IF THERE IS A BETTER ESTIMATOR FOR THE MEAN  THAN THE OBVIOUS ONE JUST OBTAINED  HOWEVER AS WILL BE SHOWN  IN SECTION REFSECCRLB THIS ONE IS DEPENDABLY THE BEST IN  THAT IT HAS THE LOWEST POSSIBLE VARIANCE FOR ANY UNBIASED ESTIMATEENDEXERCISESSECTIONMARKOV AND HIDDEN MARKOV MODELSLABELSECHMM1A HIDDEN MARKOV MODEL HMM IS A STOCHASTIC MODEL THAT IS USED TOMODEL TIMEVARYING RANDOM PHENOMENA  IT IS BASED UPON A MARKOV MODELAND CAN BE UNDERSTOOD IN TERMS OF THE STATESPACE MODELS ALREADYDERIVED  WE NOW PRESENT THE BASIC CONCEPTS PROVIDING RESOLUTION TOTHE ISSUES RAISED HERE IN CHAPTERS REFCHAPEM AND REFCHAPPATHSEARCHPLACEMENT HERE SERVES SEVERAL PURPOSES IT PROVIDES A DEMONSTRATION OFTHE UTILITY OF THE STATESPACE FORMULATION TO YET ANOTHER SYSTEM ITSMOOTHES THE DEVELOPMENT OF HMM ALGORITHMS IN LATER CHAPTERS AND ITPROVIDES INTRODUCTION AND MOTIVATION FOR TWO IMPORTANT ALGORITHMS THEEM ALGORITHM AND THE VITERBI ALGORITHM INDEXEXPECTATIONMAXIMIZATION EM ALGORITHMINDEXVITERBI ALGORITHMSUBSECTIONMARKOV MODELSBEGINFLOATINGFIGURE125ININPUTPICTUREDIRMARKOV1CAPTIONA SIMPLE MARKOV MODELLABELFIGMARKOV1ENDFLOATINGFIGUREINDEXMARKOV MODEL THE MARKOV MODEL IS USED TO MODEL THE EVOLUTIONOF RANDOM PHENOMENA THAT CAN BE IN DISCRETE STATES AS A FUNCTION OFTIME WHERE THE TRANSITION FROM ONE STATE TO THE NEXT IS RANDOMSUPPOSE THAT A SYSTEM CAN BE IN ONE OF S DISTINCT STATES AND THATAT EACH STEP OF DISCRETE TIME IT CAN MOVE TO ANOTHER STATE AT RANDOMWITH THE PROBABILITY OF THE TRANSITION AT TIME T DEPENDENT ONLY UPONTHE STATE OF THE SYSTEM AT TIME T  IT IS CONVENIENT TO REPRESENTTHIS CONCEPT USING A PROBABILISTIC STATE DIAGRAM AS SHOWN IN FIGUREREFFIGMARKOV1  IN THIS FIGURE THE MARKOV MODEL HAS THREE STATESFROM STATE 1 TRANSITIONS TO EACH OF THE STATES ARE POSSIBLE FROMSTATE 1 TO STATE 1 WITH PROBABILITY 05 AND SO FORTH  LET STDENOTE THE STATE AT TIME T WHERE ST TAKES ON ONE OF THE VALUES12LDOTSS  THE INITIAL STATE IS SELECTED ACCORDING TO APROBABILITY PII PII  PS1  IQQUAD I12LDOTSSBY THE FOREGOING DESCRIPTION THE PROBABILITY OF TRANSITION DEPENDSONLY UPON THE CURRENT STATE PST1  JST  I ST1K ST2  L LDOTS  PST1 J ST  ITHIS STRUCTURE ON THE PROBABILITIES IS CALLED THE EM MARKOVPROPERTY AND THE RANDOM SEQUENCE OF STATE VALUES S0 S1S2LDOTS IS CALLED A EM  MARKOV SEQUENCE OR A EM MARKOV  CHAIN   THIS SEQUENCE IS THE OUTPUT OF THE MARKOV MODELWE CAN DETERMINE THE PROBABILITY OF ARRIVING IN THE NEXT STATE BYADDING UP ALL THE PROBABILITIES OF THE WAYS OF ARRIVING THEREBEGINEQUATIONBEGINSPLITPST1  J  PST1JST1 PST1    QQUAD PST1  JST2 PST  2  CDOTS    QQUAD    PST1  JST  S PST  SLABELEQMARKOVP1 ENDSPLITENDEQUATIONTHE COMPUTATION IN REFEQMARKOVP1 CAN BE EXPRESSED CONVENIENTLYUSING MATRIX NOTATION  LET PBFT  BEGINBMATRIX PST  1  PST  2  VDOTS   PST  S ENDBMATRIXBE THE VECTOR OF PROBABILITIES FOR EACH STATE AND LET THE MATRIX ACONTAIN THE TRANSITION PROBABILITIESBEGINEQUATIONA  BEGINBMATRIX P11  P12  CDOTS  P1S P21  P22  CDOTS  P2S  VDOTS PS1  PS2  CDOTS  PSS ENDBMATRIXLABELEQHMMAENDEQUATIONWHERE PIJ IS AN ABBREVIATION FOR PST1ISTJ OR AIJ PST1ISTJ  FOR EXAMPLE FOR THE MARKOV MODEL OF FIGUREREFFIGMARKOV1BEGINEQUATION A  BEGINBMATRIX532  207 371  ENDBMATRIXLABELEQHMMAMATENDEQUATIONA EM STEADYSTATE PROBABILITY ASSIGNMENT IS ONE THAT DOES NOTCHANGE FROM ONE TIME STEP TO THE NEXT SO THE PROBABILITY MUST SATISFYTHE EQUATION A PBF  PBF  THIS IS A PARTICULAR EIGENEQUATIONWITH AN EIGENVALUE OF 1  MORE WILL BE SAID ABOUT EIGENVALUE PROBLEMSIN CHAPTER REFCHAPEIGENBY THE LAW OF TOTAL PROBABILITY EACH COLUMN OF A MUST SUM TO 1BEGINDEFINITION  AN MATSIZEMM MATRIX P SUCH THAT SUMJ1M PIJ 1  EACH ROW SUMS TO 1 AND EACH ELEMENT OF P IS NONNEGATIVE IS  CALLED A BF STOCHASTIC MATRIX  IF THE ROWS AND COLUMNS EACH SUM  TO 1 THEN P IS BF DOUBLY STOCHASTIC INDEXSTOCHASTIC MATRIXENDDEFINITION SEE EXERCISE REFEXSTOCHEIGTHE MATRIX A OF REFEQHMMA IS THE TRANSPOSE OF A STOCHASTIC MATRIXTHE VECTOR PIBF CONTAINS THE INITIAL PROBABILITIES  THUS WE CANWRITE THE PROBABILISTIC UPDATE EQUATION AS PBFT1  APBFTQQUAD TEXTWITH QQUAD PBF0  PIBFOR TO PUT IT ANOTHER WAYBEGINEQUATION PBFT1  APBFT  PIBF DELTATLABELEQMARKOV1ENDEQUATIONWITH PBFT  ZEROBF FOR T LEQ 0  THE SIMILARITY OFREFEQMARKOV1 TO THE FIRST EQUATION OF REFEQSTATE2 SHOULDBE APPARENT  IN COMPARING THESE TWO IT SHOULD BE NOTED THAT THESTATE REPRESENTED BY REFEQMARKOV1 IS ACTUALLY THE VECTOR OFPROBABILITIES PBFT NOT THE STATE OF THE MARKOV SEQUENCE STSUBSECTIONHIDDEN MARKOV MODELSTHE IDEA BEHIND THE HMM CAN BE ILLUSTRATED USING THE URN PROBLEMS OFELEMENTARY PROBABILITY AS SHOWN IN FIGURE REFFIGMARKOV2  SUPPOSEWE HAVE S DIFFERENT URNS EACH OF WHICH CONTAINS ITS OWN SET OFCOLORED BALLS  AT EACH INSTANT OF TIME AN URN IS SELECTED AT RANDOMACCORDING TO THE STATE IT WAS IN AT THE PREVIOUS INSTANT OF TIMETHAT IS ACCORDING TO A MARKOV MODEL  THEN A BALL IS DRAWN ATRANDOM FROM THE URN SELECTED AT TIME T  THE BALL IS WHAT WE OBSERVEAS THE OUTPUT AND THE ACTUAL STATE IS HIDDEN  THROUGHOUT THEREMAINDER OF THIS INTRODUCTION WE WILL CONTINUE TO DEVELOP THENOTATION FOR HMMS WITH DISCRETE OUTPUTS URN PROBLEMS BUT THEDEVELOPMENTS OF LATER CHAPTERS LIFT THIS RESTRICTIONBEGINFIGURECENTERINGINPUTPICTUREDIRMARKOV2CAPTIONTHE CONCEPT OF A HIDDEN MARKOV MODELLABELFIGMARKOV2ENDFIGUREINDEXHIDDEN MARKOV MODEL INDEXHMMSEEHIDDEN MARKOV MODEL THE DISTINCTION BETWEEN MARKOV MODELS AND HIDDEN MARKOV MODELS CAN BEFURTHER CLARIFIED BY CONTINUING THE ANALOGY WITH THE STATESPACEEQUATIONS IN REFEQSTATE2  EQUATION REFEQMARKOV1 PROVIDESFOR THE STATE UPDATE OF THE MARKOV SYSTEM  IN MOST LINEAR SYSTEMSHOWEVER THE STATE VECTOR IS NOT DIRECTLY OBSERVABLE INSTEAD IT ISOBSERVED ONLY THROUGH THE OBSERVATION MATRIX C ASSUMING FOR THEMOMENT THAT D IS ZERO YBFT  C XBFTSO THE STATE IS HIDDEN FROM DIRECT OBSERVATION  SIMILARLY IN THE HMMWE DO NOT OBSERVE THE STATE DIRECTLY  INSTEAD EACH STATE HAS APROBABILITY DISTRIBUTION ASSOCIATED WITH IT  WHEN THE HMM MOVES INTOSTATE ST AT TIME T THE OBSERVED OUTPUT YT IS AN OUTCOME OFA RANDOM VARIABLE YT THAT IS SELECTED ACCORDING TO DISTRIBUTIONFYTSTS WHICH WE WILL REPRESENT USING THE NOTATION FYSTS  FSYTHIS IDEA IS ILLUSTRATED IN FIGURE REFFIGMARKOVFYX  IN THE URNEXAMPLE OF THE PRECEEDING PARAGRAPH THE OUTPUT PROBABILITIES DEPENDON THE CONTENTS OF THE URNS  A SEQUENCE OF OUTPUTS FROM AN HMM ISY0 Y1 Y2LDOTS  THE UNDERLYING STATE INFORMATION IS NOTSEEN DIRECTLY IT IS HIDDENBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRHMM  ENDCENTER  CAPTIONAN HMM WITH FOUR STATESAN HMM WITH FOUR STATES    SHOWING THE STATES THE DISTRIBUTION IN EACH STATE AND    PROBABILISTIC TRANSITIONS BETWEEN STATES  LABELFIGMARKOVFYXENDFIGURETHE PROBABILITY DISTRIBUTION IN EACH STATE CAN BE OF ANY TYPE AND INGENERAL EACH STATE COULD HAVE ITS OWN TYPE OF DISTRIBUTION  MOSTOFTEN IN PRACTICE HOWEVER EACH STATE HAS THE SAME TYPE OFDISTRIBUTION BUT WITH DIFFERENT PARAMETERSLET M DENOTE THE NUMBER OF POSSIBLE OUTCOMES FROM ALL OF THE STATESAND LET YT BE THE RANDOM VARIABLE OUTPUT AT TIME T WITH OUTCOMEYT  WE CAN DETERMINE THE PROBABILITY OF EACH POSSIBLE OUTPUT BYADDING UP ALL THE PROBABILITIESBEGINALIGNED PYT  J  PYTJST  1PST1   QQUAD PYTJST2PST2  CDOTS   QQUAD  PYTJSTSPSTSENDALIGNEDLET QBFT  BEGINBMATRIXPYT1  PYT 2  VDOTS   PYT  M ENDBMATRIXAND  C  BEGINBMATRIX PYT1ST1  CDOTS  PYT1ST  S PYT2ST1  CDOTS  PYT2ST  S VDOTS PYTMST1  CDOTS  PYTMST  S ENDBMATRIXSO CIJ  PYT  IST  JFOR THE URNS SHOWN IN FIGURE REFFIGMARKOV2 WITH THE BALL COLORSBLACK GREEN AND RED CORRESPONDING TO VALUES 1 2 AND 3RESPECTIVELY  C  BEGINBMATRIX12  13    13 13   715   13 16   15    13 ENDBMATRIXEACH OF THE COLUMNS MUST SUM TO ONE  THE OUTPUT PROBABILITIESCAN BE COMPUTED BY QBFT  C PBFTTHE SIMILARITY WITH REFEQSTATE2 SHOULD BE CLEAR  BASED ON THISDISCUSSION THE HMM PARAMETERS ARE DESCRIBED BY THE TRIPLEAPIBFC MUCH LIKE OUR STATESPACE MODELSINDEXPATTERN RECOGNITION INDEXSPEECH PROCESSING THE HMMCAN BE APPLIED TO PATTERN RECOGNITION WHERE THE PATTERNS OCCUR ASEVENTS OCCURRING SEQUENTIALLY IN TIME  PERHAPS THE MOST SUCCESSFULAPPLICATION IS TO SPEECH PROCESSING  EACH WORD OR SOUND PHONEME TOBE RECOGNIZED IS REPRESENTED BY AN HMM WHERE THE OUTPUT IS SOMEFEATURE VECTOR THAT IS DERIVED FROM THE SPEECH DATA  THE RANDOMVARIABILITY IN THE FEATURE VECTOR AND THE AMOUNT OF TIME EACH FEATUREIS PRODUCED IS MODELED BY THE HMM  THE VARIABILITY IN THE DURATION OFTHE WORD IS MODELED BY THE MARKOV MODEL  THE VARIABILITY IN THEOUTPUTS IS MODELED BY THE RANDOM SELECTION FROM WITHIN EACH STATEFOR EXAMPLE IN A SMALL VOCABULARY SYSTEM WITH N WORDS THERE AREN HMMS AI PIBFI CI EACH BEING TRAINED OR ADAPTED TO REPRESENT THE PARAMETERS FOR THAT WORD  THIS IS THE TRAINING PHASE OFTHE PATTERN RECOGNITION PROBLEM  INDEXTRAINING PHASETO PERFORM RECOGNITION OF AN UNKNOWN WORD ITS SEQUENCE OF FEATUREVECTORS IS COMPUTED AND THE LIKELIHOOD PROBABILITY THAT THISSEQUENCE OF FEATURE VECTORS WAS PRODUCED BY THE HMM AI PIBFICI IS COMPUTED FOR EACH I  THAT HMM WHICH PRODUCES THE HIGHESTPROBABILITY SELECTS THE RECOGNIZED WORDTHE HMM HAS ALSO BEEN APPLIED TO HANDWRITING RECOGNITION SPEAKERIDENTIFICATION AND OTHER AREASTHE PATTERN RECOGNITION APPLICATIONIS DIAGRAMMED IN FIGURE REFFIGHMMPATRECBASED ON THIS SIMPLE DISCUSSION THERE ARE SEVERAL QUESTIONS THATCAN BE POSED IN CONJUNCTION WITH HMMSBEGINENUMERATEITEM HOW CAN THE PARAMETERS APIBFC BE ESTIMATED BASED UPON  OBSERVATIONS OF THE DATA  OR MORE GENERALLY HOW CAN THE  PARAMETERS OF OTHER OUTPUT DISTRIBUTIONS BE COMPUTED  IN OTHER  WORDS HOW CAN WE TRAIN THE PARAMETERS OF THE MODELS IN THE PATTERN  RECOGNITION PROBLEM INDEXPARAMETER ESTIMATIONITEM SUPPOSE WE HAVE AN HMM AND WE OBSERVE A SEQUENCE OF DATA  HOWCAN WE DETERMINE HOW WELL THE DATA FITS THE MODEL  IN OTHERWORDS CAN WE EFFICIENTLY DETERMINE THE LIKELIHOOD OF THE DATAINDEXMAXIMUM LIKELIHOODITEM RELATED SOMEWHAT TO THE PREVIOUS SUPPOSE WE HAVE AN HMM AND WE  OBSERVE SOME DATA SUPPOSEDLY GENERATED FROM IT  HOW CAN WE  DETERMINE THE SEQUENCE OF STATES OF THE UNDERLYING MARKOV MODEL  THAT IS WE WANT TO UNCOVER THE HIDDEN STATESENDENUMERATETHESE ISSUES ARE EXPLORED IN CHAPTERS REFCHAPEM ANDREFCHAPPATHSEARCH WHERE THE EM ALGORITHM AND THE VITERBIALGORITHM ARE INTRODUCED AND APPLIED TO THIS PROBLEMBEGINEXERCISESITEM  WRITE A SC MATLAB FUNCTION TT GENMARKOVNABPIIN WHICH  GENERATES N OUTPUTS OF A HIDDEN MARKOV MODEL WITH STATE TRANSITION  MATRIX A AND OUTPUT MATRIX B WITH INITIAL PROBABILITIES  PIINITEM FOR THE STATETRANSITION PROBABILITY MATRIX A GIVEN IN  REFEQHMMAMAT SHOW THAT THE PROBABILITY VECTOR PBF   05720 05441 06138T SATISFIES A PBF  PBFTHIS IS THE EM STEADYSTATE PROBABILITY OF THE MARKOV MODELENDEXERCISESINPUTHOMEDIRINTROPARTPROOFSINPUTHOMEDIRINTROPARTBERLMASSYSETEXSECTREFSECLTIBEGINEXERCISESITEM COMPLEX ARITHMETIC THIS EXERCISE GIVES A BRIEF REFRESHER ON  COMPLEX MULTIPLICATION AS WELL AS MATRIX MULTIPLICATION  LET Z1   AJB AND Z2  C  JD BE TWO COMPLEX NUMBERS  LET Z3  Z1  Z2  EJF  BEGINENUMERATE  ITEM SHOW THAT THE PRODUCT CAN BE WRITTEN AS BEGINBMATRIXE  F  ENDBMATRIX  BEGINBMATRIXCD  D  CENDBMATRIX BEGINBMATRIX A  B ENDBMATRIXIN THIS FORM FOUR REAL MULTIPLIES AND TWO REAL ADDS ARE REQUIREDITEM SHOW THAT THE COMPLEX PRODUCT CAN ALSO BE WRITTEN AS E  ABD  ACD QQUAD F  ABD  BCDIN THIS FORM ONLY THREE REAL MULTIPLICATIONS AND FIVE REAL ADDITIONSARE REQUIRED  IF ADDITION IS SIGNIFICANTLY EASIER THANMULTIPLICATION IN HARDWARE THEN THIS SAVES COMPUTATIONSITEM SHOW THAT THIS MODIFIED SCHEME CAN BE EXPRESSED IN MATRIX  NOTATION AS BEGINBMATRIXE  F ENDBMATRIX  BEGINBMATRIX101  011  ENDBMATRIX BEGINBMATRIXCD00  0CD  0  00D  ENDBMATRIX BEGINBMATRIX10  01  11 ENDBMATRIX  BEGINBMATRIXA  B ENDBMATRIXENDENUMERATEITEM SHOW THAT REFEQPFEZT FOR THE PARTIAL FRACTION EXPANSION OF  A ZTRANSFORM WITH REPEATED ROOTS IS CORRECTITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF THE FOLLOWINGBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2TEXTC  HZ  FRAC2  3Z113 Z12 TEXTD  HZ  FRAC56Z113 Z1214Z1ENDARRAYCHECK YOUR RESULTS USING TT RESIDUEZ IN SC MATLABITEM INVERSES OF HIGHERORDER MODES  BEGINENUMERATE  ITEM PROVE THE FOLLOWING PROPERTY FOR ZTRANSFORMS  IF XT LEFTRIGHTARROW XZTHEN TXT LEFTRIGHTARROW Z FRACD XZDZITEM USING THE FACT THAT PT UT LEFTRIGHTARROW 11PZ1  SHOW THAT T PT UT LEFTRIGHTARROW FRACPZ11PZ12ITEM DETERMINE THE ZTRANSFORM OF T2 PT UTITEM BY EXTRAPOLATION DETERMINE THE ORDER OF THE POLE OF A MODE OF  THE FORM TK PT UT  ENDENUMERATEEXSKIPITEM SHOW THAT THE AUTOCORRELATION FUNCTION DEFINED IN  REFEQAUTOCORRDEF HAS THE PROPERTY THAT  RYYK  RBARYYKITEM SHOW THAT REFEQMAAUTOCORR IS CORRECTITEM FOR THE MA PROCESS YT  FT  2FT1  3FT2WHERE FT IS A ZEROMEAN WHITENOISE RANDOM PROCESS WITHSIGMAF2  1 DETERMINE THE MATSIZE33 AUTOCORRELATIONMATRIX RITEM FOR THE FIRSTORDER REAL AR PROCESS YT1  A1 YT   FT1WITH A11 AND EFT  0 SHOW THATBEGINEQUATIONSIGMAY2  EY2T  FRACSIGMAF21A12LABELEQFIRSTARVARENDEQUATIONITEM FOR AN AR PROCESS REFEQAR2 DRIVEN BY A WHITENOISE  SEQUENCE FT WITH VARIANCE SIGMAF2 SHOW THATBEGINEQUATIONSIGMAF2  SUMI0P AI RYYI HAYKIN P 120LABELEQARINPUTVARENDEQUATIONITEM LET YT  7YT1  12 YT2  FT WHERE FT IS A  ZEROMEAN WHITENOISE RANDOM PROCESS WITH SIGMAF2  2    BEGINENUMERATE  ITEM WRITE  THE YULEWALKER EQUATIONS FOR Y  ITEM DETERMINE RYY1 AND RYY2  ITEM FIND SIGMAY2  ENDENUMERATEITEM SECONDORDER AR PROCESSES CONSIDER THE SECONDORDER REAL AR  PROCESS INDEXAUTOREGRESSIVESECONDORDERBEGINEQUATION YT2  A1 YT1  A2 YT  FT2 LABELEQYULEWALKER21ENDEQUATIONWHERE FT IS A ZEROMEAN WHITENOISE SEQUENCE  THE DIFFERENCEEQUATION IN REFEQAR3 HAS A CHARACTERISTIC EQUATION WITH ROOTS P1 P2  FRAC12A1 PM SQRTA12  4A2BEGINENUMERATEITEM USING THE YULEWALKER EQUATIONS SHOW THAT IF THE  AUTOCORRELATION VALUES  RYYLK  EYTKYBARTLARE KNOWN THEN THE MODEL PARAMETERS MAY  BE DETERMINED FROMBEGINEQUATIONBEGINSPLITA1  FRACRYY1RYY0  RYY2 RYY20   RYY21 A2  FRACRYY0RYY2  RYY21 RYY20   RYY21ENDSPLITLABELEQYW5ENDEQUATIONITEM ON THE OTHER HAND IF SIGMAY2  RYY0 AND A1 AND  A2 ARE KNOWN SHOW THAT THE AUTOCORRELATION VALUES CAN BE  EXPRESSED AS  BEGINEQUATION    LABELEQYW6BEGINSPLIT    RYY1  FRACA11A2 SIGMAY2RYY2  SIGMAY2LEFT FRACA121A2  A2RIGHTENDSPLITENDEQUATIONITEM USING REFEQARINPUTVAR AND THE RESULTS OF THIS PROBLEM  SHOW THAT  BEGINEQUATION    LABELEQYW7    RYY0  SIGMAY2  LEFTFRAC1A21A2RIGHT    FRACSIGMAF21A22  A12  ENDEQUATIONITEM USING RYY0  SIGMAY2 AND RYY1  A1  SIGMAY21A2 AS INITIAL CONDITIONS FIND AN EXPLICIT SOLUTION  TO THE YULEWALKER DIFFERENCE EQUATION RYYK  A1 RYYK1  A2 RYYK2  0IN TERMS OF P1 P2 AND SIGMAY2ENDENUMERATE HAYKIN P 121ITEM FOR THE SECONDORDER DIFFERENCE EQUATION YT2  7 YT1 12 YT  FT2WHERE FT IS A ZEROMEAN WHITE SEQUENCE WITH SIGMAF2  1DETERMINE SIGMAY2  RYY0 RYY1 AND RYY2ITEM A RANDOM PROCESS YT HAVING ZEROMEAN AND MATSIZEMM  AUTOCORRELATION MATRIX R IS APPLIED TO AN FIR FILTER WITH IMPULSE  RESPONSE VECTOR HBF  H0H1H2LDOTSHM1T  DETERMINE  THE AVERAGE POWER OF THE FILTER OUTPUT XT  EXSKIPITEM PLACE THE FOLLOWING INTO STATE VARIABLE FORM CONTROLLER  CANONICAL FORM AND DRAW A REALIZATIONBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2ENDARRAY ITEM DETERMINE THE FIRST FOUR NONZERO MARKOV PARAMETERS OF THE   SYSTEMS IN THE PREVIOUS EXERCISEITEM IN ADDITION TO THE BLOCK DIAGRAM SHOWN IN FIGURE  REFFIGTRANSFER2 THERE ARE MANY OTHER FORMS  THIS PROBLEM  INTRODUCES ONE OF THEM THE EM OBSERVER CANONICAL FORM INDEXOBSERVER CANONICAL FORM  BEGINENUMERATE  ITEM SHOW THAT THE ZTRANSFORM RELATION IMPLIED BY REFEQARMA    CAN BE WRITTEN ASBEGINEQUATIONBEGINSPLIT YZ   BBAR0 FZ  BBAR1 FZ  ABAR1 YZZ1  BBAR2 FZ  ABAR2 YZ Z2   CDOTS  BBARP FZ  ABARP YZZ1ENDSPLITLABELEQBLOCK2ENDEQUATIONITEM DRAW A BLOCK DIAGRAM REPRESENTING REFEQBLOCK2 CONTAINING  P DELAY ELEMENTS ITEM LABEL THE OUTPUTS OF THE DELAY ELEMENTS FROM RIGHT TO LEFT AS  X1 X2 LDOTS XP  SHOW THAT THE SYSTEM CAN BE PUT INTO STATE  SPACE FORM WITH A  BEGINBMATRIX ABAR1  1  0  CDOTS  0 ABAR2  0  1  CDOTS  0 VDOTS ABARP1  0  0 CDOTS  1 ABARP  0  0  CDOTS  0 ENDBMATRIXQQUAD BBF  BEGINBMATRIX BBAR1  ABAR1 BBAR0  BBAR2  ABAR2 BBAR0 CDOTS BBARP1  ABARP1 BBAR0 BBARP  ABARP BBAR0 ENDBMATRIXQQUAD CBF  BEGINBMATRIX 1  0  0  VDOTS  0 ENDBMATRIXQQUAD D  BBAR0A MATRIX A OF THIS FORM IS SAID TO BE IN EM SECOND COMPANION  FORM INDEXSECOND COMPANION FORMITEM DRAW THE BLOCK DIAGRAM IN OBSERVER CANONICAL FORM FOR HZ  FRAC 2  3Z1  4 Z21  Z1  6Z2  7 Z3AND DETERMINE THE SYSTEM MATRICES ABBF CBFT DENDENUMERATEITEM ANOTHER BLOCK DIAGRAM REPRESENTATION IS BASED UPON THE PARTIAL  FRACTION EXPANSION  ASSUME INITIALLY THAT THERE ARE NO REPEATED ROOTS SO THAT HZ  SUMK1P FRACNK1PK Z1  BEGINENUMERATEITEM DRAW A BLOCK DIAGRAM REPRESENTING THE PARTIAL FRACTION EXPANSION  BY USING THE FACT THAT FRACYZFZ  FRAC11P Z1 HAS THE BLOCK DIAGRAM BEGINCENTERINPUTPICTUREDIRTRANSFER5LATEXINPUTPICTUREDIRTRANSFER5ENDCENTERITEM LET XI I12LDOTS P DENOTE THE OUTPUTS OF THE DELAY  ELEMENTS  SHOW THAT THE SYSTEM CAN BE PUT INTO STATESPACE FORM  WITH  A  BEGINBMATRIX P1  0  0  CDOTS 00P2  0  CDOTS  0  VDOTS 0  0  0  CDOTS  PP ENDBMATRIXQQUAD BBF  BEGINBMATRIX 1  1  VDOTS  1 ENDBMATRIXQQUAD CBF  BEGINBMATRIX  N1  N2  VDOTS  NP ENDBMATRIXQQUAD D  B0A MATRIX A IN THIS FORM IS SAID TO BE A EM DIAGONALMATRIX INDEXDIAGONAL MATRIXITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF  HZ  FRAC 1  2Z1 1  5 Z1  06 Z2AND DRAW THE BLOCK DIAGRAM BASED UPON IT  DETERMINE ABBF CBFDITEM WHEN THERE ARE REPEATED ROOTS THINGS ARE SLIGHTLY MORE  COMPLICATED  CONSIDER FOR SIMPLICITY A ROOT APPEARING ONLY  TWICE  DETERMINE THE PARTIAL FRACTION EXPANSION OF HZ  FRAC1Z11 2 Z115Z12BE CAREFUL ABOUT THE REPEATED ROOT  ITEM DRAW THE BLOCK DIAGRAM CORRESPONDING TO HZ IN PARTIAL  FRACTION FORM USING ONLY THREE DELAY ELEMENTS  ITEM SHOW THAT THE STATE VARIABLES CAN BE CHOSEN SO THAT A  BEGINBMATRIX 5 00 15  0 0  0  2  ENDBMATRIXA MATRIX IN THIS FORM BLOCKS ALONG THE DIAGONAL EACH BLOCK BEINGEITHER DIAGONAL OR DIAGONAL WITH ONES IN IT AS SHOWN IS IN EM JORDANFORM INDEXJORDAN FORMENDENUMERATEITEM SHOW THAT THE SYSTEM IN REFEQSTATE3 HAS THE SAME TRANSFER  FUNCTION AND SOLUTION AS DOES THE SYSTEM IN REFEQSTATE2ITEM LABELEXSTATEOUT FOR A SYSTEM IN STATESPACE REPRESENTATION  BEGINENUMERATE  ITEM SHOW BY INDUCTION THAT REFEQXNDT1 IS CORRECT  ITEM FOR A TIMEVARYING SYSTEM AS IN REFEQSTATE4 DETERMINE    A REPRESENTATION SIMILAR TO REFEQXNDT1  ENDENUMERATEITEM INTERCONNECTION OF SYSTEMS INDEXINTERCONNECTION OF SYSTEMS  CITEKAILATH80 LET A1BBF1CBF1T AND   A2BBF2CBF2T BE TWO SYSTEMS  DETERMINE THE SYSTEM  ABBFCBFT OBTAINED BY CONNECTING THESE TWO SYSTEMS  BEGINENUMERATE  ITEM IN SERIES  ITEM IN PARALLEL  ITEM IN A FEEDBACK CONFIGURATION WITH A1BBF1CBF1T IN    THE FORWARD LOOP AND A2BBF2CBF2T IN THE FEEDBACK LOOP  ENDENUMERATEITEM SHOW THAT  BEGINBMATRIX A A1  0  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  ZEROBF ENDBMATRIX QQUAD CBFT QBFTAND BEGINBMATRIX A 0  A1  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  QBF ENDBMATRIX QQUAD CBFT ZEROBFAND ABBFCBFT ALL HAVE THE SAME TRANSFER FUNCTION FOR ALLVALUES OF A1 A2 AND QBF THAT LEAD TO VALID MATRIXOPERATIONS  CONCLUDE THAT REALIZATIONS CAN HAVE DIFFERENT NUMBERS OF STATESEXSKIPITEM CONSIDER THE SYSTEM FUNCTION HZ  FRACZ3  3Z2  2Z Z3  10Z2  31 Z  30BEGINENUMERATEITEM DRAW THE BLOCK DIAGRAM IN CONTROLLER CANONICAL FORMITEM DRAW THE BLOCK DIAGRAM IN JORDAN FORM DIAGONAL  FORM INDEXJORDAN FORMITEM HOW MANY MODES ARE REALLY PRESENT IN THE SYSTEM  THE PROBLEM  HERE IS THAT A EM MINIMAL REALIZATION OF A IS NOT OBTAINED  DIRECTLY FROM THE HZ AS GIVENENDENUMERATEITEM CITEKAILATH80 IF ABBFCBFTD WITH D NEQ 0  DESCRIBES A SYSTEM HS IN STATESPACE FORM SHOW THAT A  BBF CBFTD BBFD CBFTD 1DDESCRIBES A SYSTEM WITH SYSTEM FUNCTION 1HSITEM LABELEXUPDATEDEQ STATESPACE SOLUTIONS  BEGINENUMERATE  ITEM SHOW THAT REFEQSTATEUPDATE IS A SOLUTION TO THE    DIFFERENTIAL EQUATION IN REFEQXNCT1 FOR CONSTANT    ABCDITEM SHOW THAT AN UPDATE FROM XBFTAU TO XBFT IS AS GIVEN  IN REFEQSTATEUPDATEITEM SHOW THAT REFEQXBFT3 IS A SOLUTION TO THE DIFFERENTIAL  EQUATION IN REFEQXNCT1 FOR NONCONSTANT ABCD PROVIDED  THAT PHI SATISFIES THE PROPERTIES GIVEN  ENDENUMERATEITEM FIND A SOLUTION TO THE DIFFERENTIAL EQUATION DESCRIBED BY THE  STATESPACE EQUATIONSBEGINALIGNED XBFDOTT  BEGINBMATRIX 01  10 ENDBMATRIX XBFT EXMATSPYT  1 0 XBFTENDALIGNEDWITH XBF0  XBF0  THESE EQUATIONS DESCRIBE SIMPLE HARMONICMOTIONITEM CONSIDER THE SYSTEM DESCRIBED BYBEGINALIGNED XBFDOTT  BEGINBMATRIX 2  0  1  1 ENDBMATRIX XBFT BEGINBMATRIX 2  1 ENDBMATRIX FT EXMATSPYT  0  2 XBFTENDALIGNEDBEGINENUMERATEITEM DETERMINE THE TRANSFER FUNCTION HSITEM FIND THE PARTIAL FRACTION EXPANSION OF HSITEM VERIFY THAT THE MODES OF HS ARE THE SAME AS THE EIGENVALUES  OF AENDENUMERATEITEM VERIFY REFEQGEOM1 BY LONG DIVISION LABELGEOMETRIC    SERIESEXSKIPITEM SYSTEM IDENTIFICATION INDEXSYSTEM IDENTIFICATIONVIA BODE    PLOTS INDEXBODE PLOTFOR SYSTEM IDENTIFICATION IN THIS  EXERCISE YOU WILL DEVELOP A TECHNIQUE FOR IDENTIFICATION OF THE  PARAMETERS OF A CONTINUOUSTIME SECONDORDER SYSTEM BASED UPON  FREQUENCY RESPONSE MEASUREMENTS BODE PLOTS  ASSUME THAT THE  SYSTEM TO BE IDENTIFIED HAS AN OPENLOOP TRANSFER FUNCTION HOS  FRACBSSA  BEGINENUMERATE  ITEM SHOW THAT WITH THE SYSTEM IN A FEEDBACK CONFIGURATION AS SHOWN    IN FIGURE REFFIGBODEID1 THE TRANSFER FUNCTION CAN BE WRITTEN    AS HCS  FRACYSFS  FRAC11ABS  1BS2 BEGINFIGUREHTBP   BEGINCENTER     LEAVEVMODE     INPUTPICTUREDIRFEEDBACK1     CAPTIONSIMPLE FEEDBACK CONFIGURATION     LABELFIGBODEID1   ENDCENTER ENDFIGUREITEM SHOW THAT FRAC1HCJOMEGA  AJOMEGA ANGLE PHIJOMEGAWHERE AJOMEGA  FRAC1BSQRTBOMEGA22  AOMEGA2 QQUADTEXTANDQQUAD TAN PHIJOMEGA  FRACAOMEGAB  OMEGA2THE QUANTITIES AJOMEGA AND PHIJOMEGA CORRESPOND TO THERECIPROCAL AMPLITUDE AND THE PHASE DIFFERENCE BETWEEN INPUT ANDOUTPUTITEM SHOW THAT IF AMPLITUDEPHASE MEASUREMENTS ARE MADE AT N  DIFFERENT FREQUENCIES OMEGA1ALLOWBREAK OMEGA2 ALLOWBREAK  LDOTS ALLOWBREAK OMEGAN THEN  THE UNKNOWN PARAMETERS A AND B CAN BE ESTIMATED BY SOLVING THE  OVERDETERMINED SET OF EQUATIONS BEGINBMATRIX AJOMEGA1  OMEGA1 SQRT1 1TAN2 PHIJOMEGA1 TAN PHIJOMEGA1  OMEGA1 AJOMEGA2  OMEGA2 SQRT1 1TAN2 PHIJOMEGA2 TAN PHIJOMEGA2  OMEGA2 VDOTS AJOMEGAN  OMEGAN SQRT1 1TAN2 PHIJOMEGAN TAN PHIJOMEGAN  OMEGAN ENDBMATRIXBEGINBMATRIX B  A ENDBMATRIX BEGINBMATRIX 0  OMEGA12 TANPHIJOMEGA1 0  OMEGA22 TANPHIJOMEGA2 VDOTS 0  OMEGAN2 TANPHIJOMEGAN ENDBMATRIX  ENDENUMERATEITEM VERIFY REFEQESD1  LABELEXESD1ITEM SHOW THAT  SUMTINFTYINFTY YT2  FRAC12PI INTPIPIGYYOMEGA DOMEGAHINT RECALL THE INVERSE FOURIER TRANSFORM INDEXFOURIER TRANSFORM YT FRAC12PI INTPIPI YOMEGA EJ OMEGA TDOMEGAITEM SHOW THAT UNDER THE CONDITION THAT REFEQPSDDEC IS TRUE  THE PSD SATISFIES SYOMEGA  LIMNRIGHTARROW INFTY ELEFT FRAC1N  LEFTSUMN1N YN EJOMEGA NRIGHT2 RIGHTHINT  SHOW AND USE THE FACT THAT SUMN1N SUMM1N FNM  SUMLN1N1 NLFLITEM MODAL ANALYSIS THE FOLLOWING DATA ARE MEASURED FROM A  THIRDORDER SYSTEM Y   0320002500010000022200006000120000500001ASSUME THAT THE FIRST TIME INDEX IS 0 SO THAT Y0  032BEGINENUMERATEITEM DETERMINE THE MODES IN THE SYSTEM AND PLOT THEM IN THE COMPLEX PLANEITEM THE DATA CAN BE WRITTEN AS YT  C1P1T  C2P2T  C3P3T QQUAD T GEQ 0DETERMINE THE CONSTANTS C1 C2 AND C3ITEM TO EXPLORE THE EFFECT OF NOISE ON THE SYSTEM ADD RANDOM  GAUSSIAN NOISE TO EACH DATA POINT WITH VARIANCE SIGMA2  001  THEN FIND THE MODES OF THE NOISY DATA  REPEAT SEVERAL TIMES WITH  DIFFERENT NOISE AND COMMENT ON HOW THE MODAL ESTIMATES MOVEENDENUMERATEITEM MODAL ANALYSIS  IF YT HAS TWO REAL SINUSOIDS YT  A COSOMEGA1 T  THETA1  B COSOMEGA2 T THETA2AND THE FREQUENCIES ARE KNOWN DETERMINE A MEANS OF COMPUTING THEAMPLITUDES AND PHASES FROM MEASUREMENTS AT TIME INSTANTS T1 T2LDOTS TNEXSKIPSETEXSECTREFSECMULTGAUSS  ITEM SHOW THAT R1 FROM REFEQINVCOVAR IS CORRECTITEM SHOW THAT REFEQ2GAUSS FOLLOWS FROM REFEQMULTGAUSS AND  REFEQINVCOVARITEM SUPPOSE THAT XSIM NCMUXSIGMAX2 AND N SIM  NC0SIGMAN2 ARE INDEPENDENTLY DISTRIBUTED GAUSSIAN RVS  LET Y  XNBEGINENUMERATEITEM DETERMINE THE PARAMETERS OF THE DISTRIBUTION OF YITEM IF YY IS MEASURED WE CAN ESTIMATE X BY COMPUTING THE  CONDITIONAL DENSITY FXY  DETERMINE THE MEAN AND VARIANCE OF  THIS CONDITIONAL DENSITY  INTERPRET THESE RESULTS IN TERMS OF  GETTING INFORMATION ABOUT IF  X I SIGMAN2 GG SIGMAX2  AND  II SIGMAN2 LL SIGMAX2ENDENUMERATEITEM SUPPOSE THAT X SIM NCMUX SIGMAX2  AND Y  SIMNCMUY  SIGMAY2 ARE JOINTLY DISTRIBUTED GAUSSIAN RVS  WITH CORRELATION RHO  DETERMINE THE PARAMETERS OF THE  DISTRIBUTION OF Z  A X  BYITEM IF X SIM NC01 SHOW THAT Y  SIGMA X  MUIS DISTRIBUTED AS Y NCSIGMA2MU LABELEXGENGAUSS ITEM IF X SIM NCSIGMA2MU DETERMINE EX THE EXPECTED   VALUE OF THE ABSOLUTE VALUE OF XITEM LABELEXGAUSSEST LET X1 X2 LDOTS XN BE N  INDEPENDENT OBSERVATIONS OF A GAUSSIAN RANDOM VARIABLE X WITH  UNKNOWN MEAN AND VARIANCE  WE DESIRE TO ESTIMATE THE MEAN AND  VARIANCE OF X  THE JOINT DENSITY OF N INDEPENDENT GAUSSIAN  RVS CONDITIONED ON KNOWING THE MEAN MU AND THE VARIANCE  SIGMA2 IS FX1X2LDOTSXNMU SIGMA2 FRAC12PIN2SIGMANEXPFRAC12 SIGMA2SUMI1N XI  MU2BEGINENUMERATEITEM DETERMINE A EM MAXIMUM LIKELIHOOD ESTIMATE OF MU BY  INDEXMAXIMUM LIKELIHOOD ESTIMATION MAXIMIZING THIS JOINT DENSITY  WITH RESPECT TO MU IE TAKE THE DERIVATIVE WITH RESPECT TO  MU  CALL THE ESTIMATE OF THE MEAN OBTAINED THUS MUHATITEM SINCE MUHAT IS A FUNCTION OF RANDOM VARIABLES IT IS ITSELF    A RANDOM VARIABLE  DETERMINE THE MEAN EXPECTED VALUE OF    MUHAT  AN ESTIMATE WHOSE EXPECTED VALUE IS EQUAL TO THE VALUE    IT IS ESTIMATED IS SAID TO BE EM UNBIASED INDEXUNBIASED  ITEM DETERMINE THE VARIANCE OF MUHAT    ITEM DETERMINE AN ESTIMATE FOR SIGMA2  ENDENUMERATE  IT IS NATURAL TO ASK IF THERE IS A BETTER ESTIMATOR FOR THE MEAN  THAN THE OBVIOUS ONE JUST OBTAINED  HOWEVER AS WILL BE SHOWN  IN SECTION REFSECCRLB THIS ESTIMATOR IS DEFENDABLY THE BEST IN  THAT IT HAS THE LOWEST POSSIBLE VARIANCE FOR ANY UNBIASED ESTIMATE  EXSKIP SETEXSECTREFSECHMM1ITEM A MARKOV RANDOM PROCESS INDEXMARKOV RANDOM PROCESS XT HAS  THE PROPERTY THAT PXT3  X2XT2X2 XT1X1  PXT3  X3XT2X2WHEN T3  T2  T1 THAT IS THE PROBABILITY DEPENDS ONLY UPON THEMOST RECENT CONDITIONING EVENT  WE WILL ABBREVIATE THIS USING THENOTATION FX3X2X1  FX3X2BEGINENUMERATEITEM FOR A MARKOV PROCESS SHOW THAT FX3X1X2  FX3X2FX2X1THIS IS THE PROPERTY OF CONDITIONAL INDEPENDENCE X3 IS INDEPENDENTOF X1 PROVIDED THAT THEY ARE EACH CONDITIONED ON AN INTERMEDIATEOBSERVATION X2 INDEXMARKOV RANDOM PROCESSCONDITIONAL INDEPENDENCEITEM NOW SUPPOSE THAT XT IS A GAUSSIAN RANDOM PROCESS AND ASSUME  FOR CONVENIENCE ONLY THAT IT IS ZEROMEAN  LET  RXTS  EXT XSIF XT IS ALSO MARKOV SHOW THAT RXT3T1  FRACRXT3T2RXT2T1RXT2T2HINT USE THE FACT THAT EEXT3XT1XT2  EXT3XT1AND USE THE FORMULA FOR CONDITIONAL EXPECTATION DERIVED IN REFEQFXYENDENUMERATE ITEM  WRITE A SC MATLAB FUNCTION TT GENMARKOVNABPIIN THAT   GENERATES N OUTPUTS OF A HIDDEN MARKOV MODEL WITH STATE TRANSITION   MATRIX A AND OUTPUT MATRIX B WITH INITIAL PROBABILITIES   PIINITEM  FOR THE STATETRANSITION PROBABILITY MATRIX A GIVEN IN   REFEQHMMAMAT SHOW THAT THE PROBABILITY VECTOR PBF    05720 05441 06138T SATISFIES  A PBF  PBF  THIS IS THE EM STEADYSTATE PROBABILITY OF THE MARKOV MODELITEM FOR THE STATETRANSITION PROBABILITY MATRIX A GIVEN IN  REFEQHMMAMAT FIND A PROBABILITY VECTOR PBF SUCH THAT SHOW THAT THE PROBABILITY VECTOR PBF    05720 05441 06138T SATISFIES A PBF  PBFSUCH A PROBABILITY VECTOR IS CALLED THE EM STEADYSTATE PROBABILITYOF THE MARKOV MODEL  EXSKIPSETEXSECTREFSECPROOFSITEM SHOW THAT SQRT3 IS IRRATIONALITEM SHOW THAT THERE ARE AN INFINITE NUMBER OF  PRIMES  HINT  USE A PROOF BY CONTRADICTION ASSUMING THAT THERE  ARE ONLY A FINITE NUMBER OF PRIMES  THEN BUILD A NUMBER 2CDOT 3  CDOT 5 CDOT CDOTS CDOT P  1 WHERE P IS THE ASSUMED LAST  PRIME AND SHOW THAT THIS IS NOT DIVISIBLE BY ANY OF THE LISTED PRIMESITEM USING PROOF BY CONTRADICTION SHOW THAT SQRT2 CANNOT BE A  RATIONAL NUMBER  HINT  ASSUME SQRT2  MN FOR SOME INTEGER  M AND N WHERE THE FRACTION IS EXPRESSED IN REDUCED FORM  SHOW  THAT THIS LEADS TO A CONTRADICTIONITEM SHOW THAT IF M2 IS EVEN THEN M MUST BE EVENITEM BY TRIAL AND ERROR DETERMINE A PLAUSIBLE FORMULA FOR SUMI0N 2ITHEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR THE SUM OF THE  FIRST N ODD INTEGERS 135 CDOTS  2N1THEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR SUMI1N FRAC1I2 ITHEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM SHOW BY INDUCTION FOR EVERY POSITIVE INTEGER N THAT N3   N IS DIVISIBLE BY 3ITEM THE QUANTITY BOXED BINOMNK  FRACNKNK IS THE NUMBER OF WAYS OF CHOOSING K OBJECTS OUT OF N OBJECTSWHERE N GEQ K  THE QUANTITY BINOMNK IS ALSO KNOWN AS THE EMBINOMIAL COEFFICIENT  WE READ THE NOTATION N CHOOSE K AS NCHOOSE K INDEXBINOMIAL COEFFICIENT INDEXNKBINOMNKSHOW BY INDUCTION THAT FOR 1 LEQ K LEQ NBEGINEQUATIONN1CHOOSEK  NCHOOSEK  NCHOOSEK1LABELEQXSUMBNENDEQUATIONITEM SHOW BY INDUCTION THAT FOR N GEQ 0 SUMK0N N CHOOSE K  2NITEM SHOW BY INDUCTION THATBEGINEQUATION BOXEDXYN  SUMK0N N CHOOSE K XK YNKLABELEQBINOMENDEQUATIONTHIS IMPORTANT FORMULA IS KNOWN AS THE EM BINOMIAL  THEOREM INDEXBINOMIAL THEOREMITEM PROVE THE FOLLOWING BY INDUCTION SUMK1N K2  FRACNN12N16ITEM PROVE THE FOLLOWING BY INDUCTION BOXED SUMK1N RK  FRACRN1  1R1 QQUAD R NEQ 1 INDEXGEOMETRIC SUMITEM PROVE BY INDUCTION THAT FRAC1SQRT4N1  FRAC12CDOT FRAC34 CDOT CDOTSCDOT FRAC2N32N2CDOT FRAC2N12N LEQ  FRAC1SQRT3N1FOR INTEGERS N GEQ 1KAZARINOFF 1961 P 5ITEM PROVE BY INDUCTION THAT FOR XYN IN ZBB WITH X NEQ Y   XY DIVIDES  INDEX  INDEXDIVIDESSEE  XNYN  THIS IS WRITTEN AS XY  XNYNEXSKIPSETEXSECTREFSECLFSR1ITEM PREPARE A TABLE SHOWING THE STORAGE CONTENTS AND OUTPUTS FOR THE  LFSR SHOWN IN THE ACCOMPANYING ILLUSTRATION WITH INITIAL CONDITIONS  SHOWN IN THE DELAY ELEMENTS  THE INITIAL CONDITION IS  0001 AS SHOWN  ALSO DETERMINE THE CONNECTION POLYNOMIAL  CDBEGINCENTERINPUTPICTUREDIRLFSR5LATEXINPUTPICTUREDIRLFSR5ENDCENTERITEM CONSIDER THE LFSR DESCRIBED BY THE THE POLYNOMIAL CD  1D D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAM USING BOTH THE REALIZATION SHOWN IN  FIGURE REFFIGLFSR1 AND THE REALIZATION SHOWN IN REFFIGLFSR12ITEM FOR THE INITIAL CONDITION 001 TRACE THE OPERATION OF  BOTH REALIZATIONS OF THE LFSR AND VERIFY THAT THE OUTPUT SEQUENCE OF  EACH IS THE SAME  HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM CONSIDER THE LFSR DESCRIBED BY THE THE POLYNOMIAL 1D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAM USING BOTH THE REALIZATION SHOWN IN  FIGURE REFFIGLFSR1 AND THE REALIZATION SHOWN IN REFFIGLFSR12ITEM FOR THE INITIAL CONDITION 001 TRACE THE OPERATION OF  BOTH REALIZATIONS OF THE LFSR AND VERIFY THAT THE OUTPUT SEQUENCE OF  EACH IS THE SAME  HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM GIVEN THE SEQUENCE 0001010  BEGINENUMERATE  ITEM DETERMINE THE SHORTESTLENGTH LFSR THAT COULD PRODUCE THIS    SEQUENCE PERFORMING THE COMPUTATIONS BY HAND  ITEM CHECK YOUR WORK USING ALGORITHM REFALGMASSEY IN SC      MATLAB  ENDENUMERATEITEM SHOW THAT FOR J01LDOTSN THE OUTPUT OF THE LFSR WITH  CONNECTION POLYNOMIAL CN1D AS IN REFEQBERLMASS1 WITH  A  DM1 DN AND LNM SATISFIES DJ  0 NO DISCREPANCYITEM WRITE THE OUTPUT SEQUENCE OF AN LFSR AS A POLYNOMIAL YD  Y0  Y1 D  Y2 D2  CDOTSBEGINENUMERATEITEM USING REFEQLFSR1 SHOW THAT THE JTH COEFFICIENT IN  YDCD VANISHES FOR JPP1 LDOTS WHERE DEGREECD   P  HENCE WE CAN WRITE CDYD  ZDWHERE ZD  Z0  Z1D   CDOTS  ZP1DP1THUS KNOWING ZD WE CAN FIND THE OUTPUT BY POLYNOMIAL LONG DIVISIONBEGINEQUATIONYD  FRACZDCDLABELEQLFSRDIVENDEQUATIONITEM SHOW THAT THE COEFFICIENTS OF ZD CAN BE RELATED TO THE  INITIAL CONDITIONS OF THE LFSR BY BEGINBMATRIX1 0CDOTS  0  C1  1  CDOTS  0 C2  C1  CDOTS  0 VDOTS CP1  CP2  CDOTS  C1  1 ENDBMATRIXBEGINBMATRIXY0  Y1  Y2  VDOTS  YP1 ENDBMATRIX BEGINBMATRIX Z0  Z1  Z2  VDOTS  ZP1ENDBMATRIXENDENUMERATEITEM LET CD  1D2  D3 WITH INITIAL CONTENTS Y0Y1Y2   100 DETERMINE THE FIRST SIX OUTPUTS USING POLYNOMIAL LONG  DIVISION AS IN REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE  OBTAINED DIRECTLY FROM THE LFSR ITEM LET CD  1DD2 WITH INITIAL CONTENTS  Y0Y1     11  DETERMINE THE FIRST SIX OUTPUTS USING POLYNOMIAL LONG   DIVISION AS IN REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE OBTAINED   DIRECTLY FROM THE LFSRITEM DETERMINE THE SEQUENCE YI OF LENGTH SEVEN GENERATED BY  CD  1DD3 AND CALL ITS LENGTH N  THEN COMPUTE THE CYCLIC  AUTOCORRELATION FUNCTION INDEXAUTOCORRELATION RHOK  FRAC1N SUMI0N1 YI YIKWHERE YIK MEANS THAT THE SUBSCRIPT IS COMPUTED MODULO NPLOT THIS AUTOCORRELATION FUNCTIONENDEXERCISESSECTIONREFERENCESTHE LINEAR SYSTEMS THEORY PRESENTED HERE IN BROAD STROKES IS PAINTEDIN CONSIDERABLY FINER DETAIL IN CITERUGH1996 AND CITEKAILATH80OUR BRIEF INTRODUCTION TO LINEAR PREDICTION IS MORE EXTENSIVELYPRESENTED IN CITEDELLER1993HAYKIN1996 WHILE CONSIDERABLY MORE ONSPECTRUM ANALYSIS APPEARS IN CITESTOICAKAY1988MARPLE  THEAPPLICATIONS OF ADAPTIVE FILTERING HIGHLIGHTED HERE ARE DISCUSSED INDEPTH IN CITEHAYKIN1996 AND CITEWIDROW1985  THE HIDDEN MARKOVMODEL IS PRESENTED IN CITERABINER1989DELLER1993 ANDCITERABINERJUANG1993  FOR AN ENJOYABLE AND READABLE INTRODUCTIONTO PROOFS WITH A VARIETY OF SUGGESTIONS AND EXAMPLES AND SOME GOODMATHEMATICAL BACKGROUND CITEVELLEMAN IS RECOMMENDED  ATHOUGHTPROVOKING BOOK ON MATHEMATICAL THINKING IS CITEPOLYA1971MASSEYS ALGORITHM IS PRESENTED IN CITEMASSEY2  AN EXCELLENTPRESENTATION OF THE ALGORITHM IS IN CITEBLAHUT1983  THE BOOKCITEGOLOMB PROVIDES AN INTRODUCTION TO LFSRS AND THE PAPERCITESARWATE1980 AN INTERESTING DISCUSSION OF DECIMATEDINDEXDECIMATION MAXIMALLENGTH SEQUENCES INDEXMAXIMALLENGTH  SEQUENCE APPLICATIONS OF LFSRS TO SPREADSPECTRUM COMMUNICATIONSARE DISCUSSED IN CITEZIEMERPETERSONMEDSKIPIN ADDITION TO THE PRESENT BOOK THERE ARE A NUMBER OF OTHER BOOKSTHAT SHOULD BE CONSIDERED AS PART OF A STANDARD LIBRARY FOR SIGNALPROCESSORS  WE MENTION THE FOLLOWING AS USEFUL REFERENCESBEGINDESCRIPTIONITEMLINEAR ALGEBRA THE BOOKCITESTRANG IS AN EXCELLENT INTRODUCTION TO LINEAR ALGEBRA  THE BOOKCITEGVL PROVIDES EXTENSIVE DETAIL ON ALGORITHMS ASSOCIATED WITH LINEARALGEBRA  IT SHOULD BE A PART OF EVERY SIGNAL PROCESSORS LIBRARYCITEHORNJOHNSON IS A GOOD REFERENCE ON THE THEORY OF MATRICESITEMSTATISTICS A GENERAL BACKGROUND IN STATISTICS IS  CITEHOGGCRAIG1978  AN EXCELLENT RECENT SOURCE ON STATISTICAL  DECISION MAKING IS CITESCHARFL1991  ANOTHER IS  CITEPOOR1988BOOK  A COMPREHENSIVE WORK IS NOCITEVANREES68ITEMALGEBRA INTRODUCTORY BOOKS ON ALGEBRA ARE NOCITEFRALEIGH AND  NOCITEBIRKHOFFMACLANEITEMCALCULUS AND ANALYSIS A STANDARD REFERENCE IS CITEROYDEN  A  MORE INTRODUCTORY LEVEL IS NOCITEBUCK FUNCTIONAL ANALYSIS  TARGETED TOWARD ENGINEERS IS CITENAYLORSELLITEMNUMBER THEORY A GOOD STARTING POINT IS  CITENIVENZUCKERMAN  AN ENTERTAINING LOOK AT A VARIETY OF  APPLICATIONS IS CITESCHROEDER  A BOOK WITH A LITTLE MORE DEPTH  IS CITEHUAITEMOPTIMIZATION SOME EXCELLENT SOURCES ARECITEFLETCHER1980 CITELUENBERGER CITELUENBERGER1984ITEMNUMERICAL ANALYSIS THE CLASSIC WORK CITERALSTON IS STILL  EXCELLENT  A MORE RECENT WORK IS CITECHENEYENDDESCRIPTION LOCAL VARIABLES TEXMASTER TEST END COMPLETING THE SQUARECHAPTERCOMPLETING THE SQUARELABELAPPDXCTSINDEXCOMPLETING THE SQUARECOMPLETING THE SQUARE IS A SIMPLE ALGEBRAIC TECHNIQUE THAT ARISESFREQUENTLY ENOUGH IN BOTH SCALAR AND VECTOR PROBLEMS THAT IT IS WORTHILLUSTRATINGSECTIONTHE SCALAR CASELABELSECB1THE QUADRATIC EXPRESSIONBEGINEQUATIONJX   AX2  BX  CLABELEQCTS0ENDEQUATIONCAN BE WRITTEN AS AX2  FRACBAX  CIN COMPLETING THE SQUARE WE WRITE THIS AS A PERFECT SQUARE WITH ACONSTANT OFFSET  TAKING THE COEFFICIENT OF X AND DIVIDING BY 2 ITIS STRAIGHTFORWARD TO VERIFY THATBEGINEQUATIONBOXEDAX2FRACBAX  C  AXFRACB2A2  FRACB24A   CLABELEQCTS1ENDEQUATIONBY MEANS OF COMPLETING THE SQUARE WE CAN OBTAIN BOTH THE MINIMIZINGVALUE OF X AND THE MINIMUM VALUE OF JX IN REFEQCTS0EXAMINATION OF REFEQCTS1 REVEALS THAT THE MINIMUM MUST OCCURWHEN X  FRACB2AA RESULT ALSO READILY OBTAINED VIA CALCULUS  IN THIS CASE WE ALSOGET THE MINIMUM VALUE AS WELL SINCE IF X  FRACB2A THEN JMIN  C  FRACB24ABEGINEXAMPLE  WE DEMONSTRATE AN EXAMPLE OF THE THE USE OF COMPLETING THE SQUARE  FOR A PROBLEM OF ESTIMATING A GAUSSIAN RANDOM VARIABLE OBSERVED IN  GAUSSIAN NOISE  SUPPOSE X SIM NCMUXSIGMAX2 AND N SIM  NC0SIGMAN2 WHERE X IS REGARDED AS A SIGNAL AND N IS  REGARDED AS NOISE  WE MAKE AN OBSERVATION OF THE SIGNAL IN THE  NOISE Y  XNGIVEN A MEASUREMENT OF YY WE DESIRE TO FIND FXY  BY BAYESTHEOREM FXY  FRACFYXFXINTINFTYINFTY FYXFXDXTHE DENSITY FYX CAN BE OBTAINED BY OBSERVING THAT FOR A GIVENVALUE OF XX Y  XNIS SIMPLY A SHIFT OF THE RANDOM VARIABLE N AND HENCE IS GAUSSIANWITH VARIANCE SIGMAN2 AND MEAN X  THAT IS FYX  FNYXWHERE FN IS THE DENSITY OF N  WE CAN THEREFORE WRITE FXY  FRACFNXYFXINTINFTYINFTY FYXFXDXTHE CONSTANT IN THE DENOMINATOR IS SIMPLY A NORMALIZING VALUE TO MAKETHE DENSITY FXY INTEGRATE TO 1 WE WILL CALL IT C AND PAYLITTLE ATTENTION TO IT  WE CAN WRITE FXY  FRAC1C FRAC1SIGMAN SQRT2PIEYX22SIGMAN2 FRAC1SIGMAX SQRT2PIEXMUX22SIGMAX2LET US FOCUS OUR ATTENTION ON THE EXPONENT WHICH WE DENOTE BY E E  FRAC12SIGMAN2YX2 FRAC12SIGMAX2XMUX2THIS CAN BE WRITTEN AS E  X2LEFTFRAC12SIGMAN2  FRAC1SSIGMAX2RIGHT X LEFTFRACYSIGMAN2  FRACMUXSIGMAX2RIGHT C1WHERE C1 DOES NOT DEPEND UPON X  BY COMPLETING THE SQUARE WEHAVE E  FRACSIGMAX2  SIGMAN22SIGMAN2SIGMAX2LEFT X   FRACYSIGMAX2  MUX SIGMAN2SIGMAX2     SIGMAN2RIGHT  C2WHERE C2 DOES NOT DEPEND UPON X  THE DENSITY CAN THUS BE WRITTEN FXY  LEFTFRAC12PI CSIGMANSIGMAXEC2RIGHTEXPLEFTFRAC12LEFTX  FRACYSIGMAX2  MUX      SIGMAN2SIGMAX2      SIGMAN2RIGHT1LEFTSIGMAN2SIGMAX2SIGMAX2SIGMAN2RIGHTRIGHTTHIS HAS THE FORM OF A GAUSSIAN DENSITY SO THE CONSTANTS IN FRONT OFTHE EXPONENTIAL MUST BE SUCH THAT THIS INTEGRATES TO 1  THE MEAN OFTHIS GAUSSIAN DENSITY IS MUXY  FRACSIGMAX2SIGMAX2SIGMAN2 Y FRACSIGMAN2SIGMAX2  SIGMAN2MUXAND THE VARIANCE IS SIGMAXY2  FRACSIGMAN2SIGMAX2SIGMAX2SIGMAN2LET IS CONSIDER AN INTERPRETATION OF THIS RESULT  IF SIGMAN2 GGSIGMAX2 THEN AN OBSERVATION OF  Y DOES NOT TELL US MUCH ABOUTX BECAUSE THE INTERFERING NOISE N IS TOO STRONG  THE INFORMATIONWE HAVE ABOUT X GIVEN Y IS THUS ABOUT THE SAME AS THE INFORMATIONWE HAVE ABOUT X ALONE  THIS OBSERVATION IS VALIDATED IN THEANALYSIS IF SIGMAN2 GG SIGMAX2 THEN MUXY APPROX MUXAND  SIGMAXY2 APPROX SIGMAX2ON THE OTHER HAND IF THE NOISE VARIANCE IS SMALL SO THAT SIGMAN2LL SIGMAX2 THEN AN OBSERVATION OF Y IS ALMOST THE SAME AS ANOBSERVATION ON X ITSELF IN THIS CASE WE HAVE MUXY APPROX Y QQUAD TEXTANDQQUADSIGMAXY2 APPROX SIGMAN2ENDEXAMPLESECTIONTHE MATRIX CASELABELSECB2THE EQUATION  XBFT A XBF  XBFTYBF  CWHERE A IS SYMMETRIC AND INVERTIBLE CAN BE WRITTEN ASBEGINEQUATION XBFZBFT A XBFZBF  DLABELEQCTS2ENDEQUATIONWHERE ZBF  FRAC12 A1 YBFAND D  C  FRAC12YBFT ZBFSETEXSECTREFSECB1BEGINEXERCISESITEM THE CHARACTERISTIC FUNCTION INDEXCHARACTERISTIC FUNCTION OF A  INDEXCHARACTERISTIC FUNCTIONGAUSSIAN OF A RANDOM VARIABLE X IS  THE CONJUGATE OF THE FOURIER TRANSFORM OF ITS DENSITY PHIXOMEGA  INT FXX EJOMEGA XDXBEGINENUMERATEITEM SHOW THAT FOR A GAUSSIAN DENSITY WITH FXF  FRAC1SQRT2PI SIGMA  EXMU22SIGMA2HAS THE CHARACTERISTIC FUNCTION PHIXOMEGA  EXPLEFTJMU OMEGA  FRAC12OMEGA2  SIGMA2RIGHTITEM TO FOLLOW UP THE CHARACTERISTIC FUNCTION IDEA SHOW THAT THE  NTH MOMENT INDEXMOMENTS OF A RV OF X CAN BE OBTAINED FROM  ITS CHARACTERISTIC FUNCTION BY EXN  LEFTFRAC1JN FRACDN PHIXOMEGADOMEGAN  RIGHTOMEGA0INDEXCHARACTERISTIC FUNCTIONMOMENTS USINGENDENUMERATEITEM SHOW THAT THE CONDITIONAL DENSITY IN REFEQFXY IS  CORRECTEXSKIPSETEXSECTREFSECB2ITEM USING REFEQCTS2 DETERMINE THE FXBFYBF  WHEN Y  XNAND X AND N ARE GAUSSIANDISTRIBUTED RANDOM VECTORS WITH X SIM NCMUBFRX QQUADTEXTANDQQUAD   N SIM NCZEROBFRNENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONAN APPLICATION LFSRS AND MASSEYS ALGORITHMLABELSECLFSR1IN THIS SECTION WE INTRODUCE THE LINEAR FEEDBACK SHIFT REGISTER LFSRINDEXLINEAR FEEDBACK SHIFT REGISTER LFSR WHICHIS NOTHING MORE THAN A DETERMINISTIC AUTOREGRESSIVE SYSTEM  THECONCEPTS PRESENTED HERE WILL ILLUSTRATE SOME OF THE LINEAR SYSTEMSTHEORY PRESENTED IN THIS CHAPTER PROVIDE A DEMONSTRATION OF SOMEMETHODS OF PROOF AND INTRODUCE OUR FIRST ALGORITHMINPUTINTRODIRALGTEXTBOXINPUTINTRODIRGF2BOXAN LFSR IS SIMPLY AN AUTOREGRESSIVE FILTER OVER A FIELD FINDEXFIELD SEE BOX REFBOXALGEBRA THAT HAS NO INPUTSIGNAL  AN LFSR IS SHOWN IN FIGURE REFFIGLFSR1  AN ALTERNATIVEREALIZATION PREFERRED IN HIGHSPEED IMPLEMENTATIONS BECAUSE THEADDITION OPERATIONS ARE NOT CASCADED IS SHOWN IN FIGURE REFFIGLFSR12THE INTERNAL STATE SEQUENCE OF THIS ALTERNATIVE REALIZATION IS NOTNECESSARILY BUT WITH APPROPRIATE INITIAL CONDITIONS THE OUTPUTSEQUENCE IS THE SAMEIF THE CONTENTS ARE BINARY IT IS HELPFUL TO VIEW THE STORAGE ELEMENTSAS D FLIPFLOPS SO THAT THE MEMORY OF THE LFSR IS SIMPLY A SHIFTREGISTER AND THE LFSR IS A DIGITAL STATE MACHINE  FOR A BINARY LFSRTHE CONNECTIONS ARE EITHER 1 OR 0 CONNECTION OR NO CONNECTION ANDALL OPERATIONS ARE CARRIED OUT IN GF2 INDEXFIELDFINITE FIELDTHAT IS MODULO 2 SEE BOX REFBOXGF2  MASSEYS ALGORITHM APPLIESOVER ANY FIELD BUT MOST COMMONLY IT IS USED IN CONNECTION WITH THEBINARY FIELDTHE OUTPUT OF THE LFSR ISBEGINEQUATION YJ  SUMI1P CI YJIQQUAD  JPP1P2LDOTSLABELEQLFSR1ENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLFSR1    CAPTIONLFSR REALIZATION    LABELFIGLFSR1  ENDCENTERENDFIGUREBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLFSR2    CAPTIONALTERNATIVE LFSR REALIZATION    LABELFIGLFSR12  ENDCENTERENDFIGURETHE NUMBER OF FEEDBACK COEFFICIENTS P IS CALLED THE EM LENGTH OFTHE LFSRBEGINEXAMPLE LABELEXMMASS1  THE LFSR OVER GF2 SHOWN IN FIGURE REFFIGLFSR2A SATISFIES YJ  YJ1  YJ3WITH INITIAL REGISTER CONTENTS 001 THE LFSR OUTPUT SEQUENCE ISSHOWN IN FIGURE REFFIGLFSR2B WHERE THE NOTATION DZ1 ISEMPLOYED  THE ALTERNATIVE REALIZATION IS SHOWN IN FIGUREREFFIGLFSR2C WITH ITS CORRESPONDING OPERATION SHOWN IN FIGUREREFFIGLFSR2DBEGINCENTERENDCENTERAFTER J6 THE SEQUENCE REPEATS SO THAT SEVEN DISTINCT STATES OCCURIN THIS DIGITAL STATE MACHINE  NOTE THAT FOR THIS LFSR THE REGISTERCONTENTS ASSUME ALL POSSIBLE NONZERO SEQUENCES OF THREE DIGITSENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREBLOCK DIAGRAMINPUTPICTUREDIRLFSR4A INPUTPICTUREDIRLFSR4ALATEXQQUADSUBFIGUREOUTPUT SEQUENCEBEGINTABULARCCC HLINEJ  STATE  YJ OUTPUT  HLINE0  001 1 1  100 1 2  110 1 3  111 0 4  011 1 5  101 0 6  010 0 HLINE7  001 1 VDOTS  VDOTS  VDOTS HLINEENDTABULARSUBFIGUREALTERNATE BLOCK DIAGRAMINPUTPICTUREDIRLFSR4 QQUAD INPUTPICTUREDIRLFSR4LATEXSUBFIGUREOUTPUT SEQUENCE FOR ALTERNATIVEREALIZATIONBEGINTABULARCCC HLINE J  STATE  YJ OUTPUT  HLINE0  001 1 1  101 1 2  111 1 3  110 0 4  011 1 5  100 0 6  010 0 HLINE7  001 1 VDOTS  VDOTS  VDOTS HLINEENDTABULARCAPTIONA BINARY LFSR AND ITS OUTPUT    LABELFIGLFSR2  ENDCENTERENDFIGURETAKING THE ZTRANSFORM OF REFEQLFSR1 WE OBTAINBEGINEQUATION YZ1C1 Z1  C2 Z2  CDOTS  CP ZP  0LABELEQLFSR10ENDEQUATIONIT WILL BE CONVENIENT TO REPRESENT THE LFSR USING THE POLYNOMIAL INREFEQLFSR10 IN THE FORM CD  1 C1 D  C2 D2  CDOTS  CP DPWHERE D  Z1 IS A DELAY OPERATOR  WE NOTE THAT THE OUTPUTSEQUENCE PRODUCED BY THE LFSR DEPENDS UPON BOTH THE FEEDBACKCOEFFICIENTS AND THE INITIAL CONTENTS OF THE STORAGE REGISTERSSUBSECTIONISSUES AND APPLICATIONS OF LFSRSWITH A CORRECTLY DESIGNED FEEDBACK POLYNOMIAL CD THE OUTPUTSEQUENCE OF A BINARY LFSR IS A MAXIMALLENGTH SEQUENCE PRODUCING2P1 OUTPUTS BEFORE THE SEQUENCE REPEATS INDEXMAXIMALLENGTH  SEQUENCE THIS SEQUENCE ALTHOUGH NOT TRULY RANDOM EXHIBITS MANY OFTHE CHARACTERISTICS OF NOISE SUCH AS PRODUCING RUNS OF ZEROS AND ONESOF DIFFERENT LENGTHS HAVING A CORRELATION FUNCTION THAT APPROXIMATESA DELTA FUNCTION AND SO FORTH  THE SEQUENCE PRODUCED IS SOMETIMESCALLED A PSEUDONOISE SEQUENCE INDEXPSEUDONOISE SEQUENCEPSEUDONOISE SEQUENCES ARE EMPLOYED IN A VARIETY OF APPLICATIONSINCLUDING SPREADSPECTRUM COMMUNICATIONS ERROR DETECTION ANDRANGING  THE GLOBAL POSITIONING SYSTEM BASED ON AN ARRAY OFSATELLITES IN GEOSYNCHRONOUS ORBIT EMPLOYS PSEUDONOISE SEQUENCES TOCARRY TIMING INFORMATION USED FOR NAVIGATIONAL PURPOSESIN SOME OF THESE APPLICATIONS THE FOLLOWING PROBLEM ARISES GIVEN ASEQUENCE Y0ALLOWBREAK Y1ALLOWBREAK LDOTSALLOWBREAKYN1 DEEMED TO BE THE OUTPUT OF AN LFSR DETERMINE THE FEEDBACKCONNECTION POLYNOMIAL CD AND THE INITIAL REGISTER CONTENTS OF THESHORTEST LFSR THAT COULD PRODUCE THE SEQUENCE  SOLVING THIS PROBLEMIS THE FOCUS OF THE REMAINDER OF THIS SECTION  THE ALGORITHM WEDEVELOP IS KNOWN AS MASSEYS INDEXMASSEYS ALGORITHM ALGORITHMNOT ONLY DOES IT SOLVE THE PARTICULAR PROBLEM STATED HERE BUT AS WESHALL SEE IT PROVIDES AN EFFICIENT ALGORITHM FOR SOLVING A PARTICULARSET OF TOEPLITZ EQUATIONSAN LFSR THAT PRODUCES THE SEQUENCE Y0Y1 LDOTS YN1COULD CLEARLY BE OBTAINED FROM AN LFSR OF LENGTH N EACH STORAGEELEMENT CONTAINING ONE OF THE VALUES  HOWEVER THIS MAY NOT BE THESHORTEST POSSIBLE LFSR  ANOTHER APPROACH TO THE SYSTEM SYNTHESIS ISTO SET UP A SYSTEM OF EQUATIONS OF THE FOLLOWING FORM ASSUMING FORTHIS EXAMPLE THAT THE LENGTH OF THE LFSR IS P3 BEGINBMATRIX Y2  Y1  Y0 Y3 Y2  Y1 Y4Y3Y2ENDBMATRIXBEGINBMATRIXC1  C2  C3ENDBMATRIX  BEGINBMATRIXY3  Y4  Y5 ENDBMATRIXTHESE EQUATIONS ARE IN THE SAME FORM AS THE YULEWALKERINDEXYULEWALKER EQUATIONS EQUATIONS IN REFEQYW3 INPARTICULAR THE MATRIX ON THE LEFT IS A TOEPLITZ MATRIXINDEXTOEPLITZ MATRIX WHEREAS THE YULEWALKER EQUATIONS WEREORIGINALLY DEVELOPED IN THIS BOOK IN THE CONTEXT OF A STOCHASTICSIGNAL MODEL WE OBSERVE THAT THERE IS A DIRECT PARALLEL WITHDETERMINISTIC AUTOREGRESSIVE SIGNAL MODELSKNOWING THE VALUE OF P THE YULEWALKER EQUATIONS COULD BE SOLVED BYANY MEANS AVAILABLE TO SOLVE P EQUATIONS IN P UNKNOWNS  HOWEVERDIRECTLY SOLVING THIS SET OF EQUATIONS IS INEFFICIENT IN AT LEAST TWOWAYSBEGINENUMERATEITEM A GENERAL SOLUTION OF A MATSIZEPP SET OF EQUATIONS  REQUIRES OP3 OPERATIONS  WE ARE INTERESTED IN DEVELOPING AN  ALGORITHM THAT REQUIRES FEWER OPERATIONS  THE ALGORITHM WE DEVELOP  REQUIRES OP2 OPERATIONSITEM FREQUENTLY THE ORDER P IS NOT KNOWN IN ADVANCE  THE VALUE OF  P COULD BE DETERMINED BY STARTING WITH A SMALL VALUE OF P AND  INCREASING THE SIZE OF THE MATRIX UNTIL AN LFSR IS OBTAINED THAT  PRODUCES THE ENTIRE SEQUENCE  THIS COULD BE DONE WITHOUT TAKING  INTO ACCOUNT THE RESULT FOR SMALLER VALUES OF P  MORE DESIRABLE  WOULD BE AN ALGORITHM THAT BUILDS RECURSIVELY ON PREVIOUSLYOBTAINED  SOLUTIONS TO OBTAIN A NEW SOLUTION  THIS IS IN FACT HOW WE  PROCEEDENDENUMERATESINCE WE BUILD UP THE LFSR USING INFORMATION FROM PRIOR COMPUTATIONSWE NEED A NOTATION TO REPRESENT THE FEEDBACK CONNECTION POLYNOMIALUSED AT DIFFERENT STAGES OF THE ALGORITHM  LET CND  1 C1ND  CDOTS  CLNNDLNDENOTE THE FEEDBACK CONNECTION POLYNOMIAL FOR THE LFSR CAPABLE OFPRODUCING THE OUTPUT SEQUENCE  Y0 Y1 LDOTS YN1 WHERELN IS THE DEGREE OF THE FEEDBACK CONNECTION POLYNOMIALTHE ALGORITHM WE OBTAIN PROVIDES AN EFFICIENT WAY OF SOLVING THEYULEWALKER EQUATIONS WHEN P IS NOT KNOWN  IN CHAPTERREFCHAPSPECIALMAT WE ENCOUNTER AN ALGORITHM FOR SOLVING TOEPLITZMATRIX EQUATIONS WITH FIXED P THE LEVINSONDURBIN ALGORITHM  ATHIRD GENERAL APPROACH BASED UPON THE EUCLIDEAN ALGORITHM IS ALSOKNOWN SEE EG CITEBLAHUT1992  EACH OF THESE ALGORITHMS HASOP2 COMPLEXITY BUT THEY HAVE TENDED TO BE USED IN DIFFERENTAPPLICATION AREAS THE LEVINSONDURBIN ALGORITHM BEING MOST COMMONLYUSED WITH LINEAR PREDICTION AND SPEECH PROCESSING AND THE MASSEY OREUCLIDEAN ALGORITHM BEING USED IN FINITEFIELD APPLICATIONS SUCH ASERRORCORRECTION CODINGSUBSECTIONMASSEYS ALGORITHMWE BUILD THE LFSR THAT PRODUCES THE ENTIRE SEQUENCE BY SUCCESSIVELYMODIFYING AN EXISTING LFSR IF NECESSARY TO PRODUCE INCREASINGLYLONGER SEQUENCES  WE START WITH AN LFSR THAT COULD PRODUCE Y0  WEDETERMINE IF THAT LFSR COULD ALSO PRODUCE THE SEQUENCE Y0Y1IF IT CAN THEN NO MODIFICATIONS ARE NECESSARY  IF THE SEQUENCECANNOT BE PRODUCED USING THE CURRENT LFSR CONFIGURATION WE DETERMINEA NEW LFSR THAT CAN PRODUCE THE ENTIRE SEQUENCE  WE PROCEED THIS WAYINDUCTIVELY EVENTUALLY CONSTRUCTING AN LFSR CONFIGURATION THAT CANPRODUCE THE ENTIRE SEQUENCE  Y0 Y1 LDOTS YN1  BY THISPROCESS WE OBTAIN A SEQUENCE OF POLYNOMIALS AND THEIR DEGREES BEGINMATRIXC1DL1 C2DL2 VDOTS CNDLN ENDMATRIXWHERE THE LAST LFSR PRODUCES Y0LDOTSYN1AT SOME INTERMEDIATE STEP SUPPOSE WE HAVE AN LFSR CND OFDEGREE LN THAT PRODUCES Y0ALLOWBREAK Y1ALLOWBREAKLDOTSALLOWBREAK YN1 FOR SOME N  N  WE CHECK IF THIS LFSRWILL ALSO PRODUCE YN BY COMPUTING THE OUTPUT YHATN  SUMI1LN CNI YNIIF YHATN IS EQUAL TO YN THEN THERE IS NO NEED TO UPDATE THELFSR AND CN1D  CND  OTHERWISE THERE IS SOMENONZERO EM DISCREPANCY DN  YN YHATN  YN  SUMI1LN CNI YNI SUMI0LN CNI YNIIN THIS CASE WE WILL UPDATE OUR LFSR USING THE FORMULABEGINEQUATIONCN1D  CND  A DL CMDLABELEQBERLMASS1ENDEQUATIONWHERE A IS SOME ELEMENT IN THE FIELD L IS AN INTEGER ANDCMD IS ONE OF THE PRIOR LFSRS PRODUCED BY OUR PROCESS THATALSO HAD A NONZERO DISCREPANCY DM  USING THIS NEW LFSR WECOMPUTE THE NEW DISCREPANCY DENOTED BY DN ASBEGINALIGNDN SUMI0LN1 CN1I YNI  NONUMBER  SUMI0LN CNI YNI  A SUMI0LMCMI YNILLABELEQBM2ENDALIGNNOW LET LNM  THEN THE SECOND SUMMATION GIVES A SUMI0LM CMI YMI  A DMTHUS IF WE CHOOSE A  DM1 DN THEN THE SUMMATION INREFEQBM2 GIVES DN1  DN1  DM1DN DM  0SO THE NEW LFSR PRODUCES THE SEQUENCE Y0Y1LDOTSYNSUBSECTIONCHARACTERIZATION OF LFSR LENGTH IN MASSEYS ALGORITHMTHE UPDATE IN REFEQBERLMASS1 IS IN FACT THE HEART OF MASSEYSALGORITHM  FROM AN OPERATIONAL POINT OF VIEW NO FURTHER ANALYSIS ISNECESSARY  HOWEVER THE PROBLEM WAS TO FIND THE SHORTEST LFSRPRODUCING A GIVEN SEQUENCE  WE HAVE PRODUCED A MEANS OF FINDING ANLFSR BUT HAVE NO INDICATION YET THAT IT IS THE SHORTESTESTABLISHING THIS WILL REQUIRE SOME ADDITIONAL EFFORT IN THE FORM OFTWO THEOREMS  THE PROOFS ARE CHALLENGING BUT IT IS WORTH THE EFFORTTO THINK THEM THROUGHIN GENERAL CONSIDERABLE SIGNAL PROCESSING RESEARCH FOLLOWS THISGENERAL PATTERN AN ALGORITHM MAY BE ESTABLISHED THAT CAN BE SHOWN TOWORK EMPIRICALLY FOR SOME PROBLEM BUT CHARACTERIZING ITS PERFORMANCELIMITS OFTEN REQUIRES SIGNIFICANT ADDITIONAL EFFORTBEGINTHEOREM  SUPPOSE THAT AN LFSR OF LENGTH LN PRODUCES THE SEQUENCE Y0  Y1 ALLOWBREAK LDOTS ALLOWBREAK YN1 BUT NOT THE  SEQUENCE Y0Y1ALLOWBREAKLDOTS ALLOWBREAK YN  THEN ANY  LFSR THAT PRODUCES THE LATTER SEQUENCE MUST HAVE A LENGTH LN1  SATISFYING LN1 GEQ N1  LNENDTHEOREMBEGINPROOF  THE THEOREM IS ONLY OF PRACTICAL INTEREST IF LN  N  OTHERWISE IT IS TRIVIAL TO PRODUCE THE SEQUENCE  LET US TAKE  THEN LN  N  LET CND  1C1N D  CDOTS  CLNNDLNREPRESENT THE CONNECTIONS FOR THE LFSR WHICH PRODUCES Y0Y1LDOTS YN1 AND LET CN1D  1C1N1 D  CDOTS CLN1N1DLN1DENOTE THE CONNECTIONS FOR THE LFSR WHICH PRODUCES Y0Y1 LDOTSYN  NOW WE DO A PROOF BY CONTRADICTION INDEXPROOFBY  CONTRADICTION ASSUME CONTRARY TO THE THEOREM THATBEGINEQUATIONLN1 LEQ N  LNLABELEQLFSRCONTENDEQUATIONFROM THE DEFINITIONS OF THE CONNECTION POLYNOMIALS WE OBSERVE THATBEGINEQUATION SUMI1LN CIN YJI QUAD BEGINCASES  YJ  JLNLN1 LDOTS N1 NEQ YN  JNENDCASESLABELEQLFSR3ENDEQUATIONANDBEGINEQUATION  LABELEQLFSR4  SUMI1LN1 CIN1 YJI  YJ QQUAD JLN1  LN11 LDOTS NENDEQUATIONFROM REFEQLFSR4 WE HAVE YN  SUMI1LN1 CIN1 YNITHE INDICES IN THIS SUMMATION RANGE FROM N1 TO NLN1 WHICHBECAUSE OF THE CONTRARY ASSUMPTION MADE IN REFEQLFSRCONT IS ASUBSET OF THE RANGE LN LN1 LDOTS N1  THUS THE EQUALITY INREFEQLFSR3 APPLIES AND WE CAN WRITE YN  SUMI1LN1 CIN1 YNI SUMI1LN1 CN1I SUMK1LN CNK YNIKINTERCHANGING THE ORDER OF SUMMATION WE HAVEBEGINEQUATIONYN  SUMK1LN CNK SUMI1LN1 CN1IYNIK LABELEQLFSR5ENDEQUATIONSETTING JN IN REFEQLFSR3 WE OBTAIN YN NEQ SUMK1LN CKNYNKIN THIS SUMMATION THE INDICES RANGE FROM N1 TO NLN WHICHBECAUSE OF REFEQLFSRCONT IS A SUBSET OF THE RANGELN1LN11LDOTSN OF REFEQLFSR4  THUS WE CAN WRITEBEGINEQUATIONYN NEQ SUMK1LN CNK SUMI1LN1CIN1 YNKILABELEQLFSR6ENDEQUATIONCOMPARING REFEQLFSR5 WITH REFEQLFSR6 WE OBSERVE ACONTRADICTION  HENCE THE ASSUMPTION ON THE LENGTH OF THE LFSRS MUSTHAVE BEEN INCORRECTBY THIS CONTRADICTION WE MUST HAVE LN1 GEQ N  1  LNENDPROOFSINCE THE SHORTEST LFSR THAT PRODUCES THE SEQUENCE Y0Y1LDOTSYNMUST ALSO PRODUCE THE FIRST PART OF THAT SEQUENCE WE MUST HAVELN1 GEQ LN  COMBINING THIS WITH THE RESULT OF THE THEOREMWE OBTAINBEGINEQUATION LN1 GEQ MAXLN N1LNLABELEQLNP1ENDEQUATIONWE CONCLUDE THAT THE SHIFT REGISTER CANNOT BECOME SHORTER AS MOREOUTPUTS ARE PRODUCEDWE HAVE SEEN HOW TO UPDATE THE LFSR TO PRODUCE A LONGER SEQUENCE USINGREFEQBERLMASS1 AND ALSO HAVE SEEN THAT THERE IS A LOWER BOUNDON THE LENGTH OF THE LFSR  WE NOW SHOW THAT THIS LOWER BOUND CAN BEACHIEVED WITH EQUALITY THUS PROVIDING THE EM SHORTEST LFSR WHICHPRODUCES THE DESIRED SEQUENCEBEGINTHEOREM  LET LI CIDI02LDOTSN BE A  SEQUENCE OF MINIMUMLENGTH HBOXLFSRS THAT PRODUCE THE SEQUENCE  Y0Y1LDOTS YI1  IF CN1D NEQ CND  THEN A NEW LFSR CAN BE FOUND THAT SATISFIES  LET THE CONNECTION POLYNOMIAL CND PRODUCE THE SEQUENCE  Y0 Y1 LDOTS YN1  ALSO LET CM M  N  DENOTE THE EM LAST CONNECTION POLYNOMIAL BEFORE CND  WHICH CAN PRODUCE THE SEQUENCE  Y0 Y1 LDOTS YM1 BUT  NOT THE SEQUENCE  Y0Y1 LDOTS YM  LET LM AND LN  BE THE LENGTHS OF THE LFSRS DESCRIBED BY CND AND  CMD RESPECTIVELY   LET DN  DENOTE THE EM DISCREPANCY BETWEEN THE YN AND THE NTH OUTPUT  OF THE LFSR WITH CND DN  YN  SUMI1LN CIN YNIIF THE DISCREPANCY DN0 THEN CN1D  CNDOTHERWISE IF DN NEQ 0 THAT IS IF CND FAILSTO PRODUCE THE SEQUENCE  Y0Y1 LDOTS YN A NEW POLYNOMIALCN1D WILL PRODUCE THE SEQUENCE WHEREBEGINEQUATION CN1D  CND  DN DM1 DNMCMDLABELEQCUPDATEENDEQUATIONFURTHERMORE THE DEGREE OF THE NEW POLYNOMIAL SATISFIESREFEQLNP1 WITH EQUALITYLN1  MAXLNN1  LNENDTHEOREMBEGINPROOF  WE WILL DO A PROOF BY INDUCTION INDEXPROOFBY INDUCTION TAKING  AS THE INDUCTIVE HYPOTHESIS THATBEGINEQUATIONLK1  MAXLK K1LKLABELEQLUPDATEENDEQUATIONFOR K01LDOTSN  THIS CLEARLY HOLDS WHEN K0 SINCE L00  TO GET STARTED LET M1C1D  1 D1  1 AND L1  0 AN LFSR OF LENGTH0  WE ALSO ASSUME AS AN INITIAL CONDITION THAT Y1  1THE OUTPUT OF THE INITIAL LFSR IS SUMI1L1 C1I SI  0THE SUM IS EMPTY  IF Y0 IS 0 THE LFSR CORRECTLY PRODUCES THEOUTPUT AND WE SET C0D  C1D  1OTHERWISE THERE IS A EM DISCREPANCY D0 BETWEEN THEOUTPUT OF THE INITIAL LFSR AND Y0 D0  Y0 SUMI1L1 CI1 YI  Y0THEN BY REFEQCUPDATEBEGINALIGNEDC0D  C1D  D0 D  1 Y0 DENDALIGNEDTHE OUTPUT OF THIS LFSR IS SUMI11  Y01  Y0SO C0D CORRECTLY PRODUCES THE OUTPUTNOW WE TAKE AS THE INDUCTIVE HYPOTHESIS THAT THERE IS A LFSRCND SATISFYINGBEGINEQUATIONYJ  SUMI1LN CIN YJI  BEGINCASES  0  JLN LN1 LDOTS N1 DN  JNENDCASESLABELEQLFSR5ENDEQUATIONWHERE DN NEQ 0 IS THE DISCREPANCY  LET CM M  N DENOTE THE EM LAST CONNECTION POLYNOMIALBEFORE CND THAT CAN PRODUCE THE SEQUENCE  Y0 Y1LDOTS YM1 BUT NOT THE SEQUENCE  Y0Y1 LDOTS YMSUCH THAT LM  LN LM  LN THEN LM1  LNHENCE IN LIGHT OF REFEQLUPDATEBEGINEQUATION LM1  LN  M1LMLABELEQLUPDATE2ENDEQUATIONIF CN1D IS UPDATED FROM CND ACCORDING TOREFEQBERLMASS1 WITH LNM WE HAVE ALREADY OBSERVED THAT ITIS CAPABLE OF PRODUCING THE SEQUENCE Y0Y1LDOTSYN  BY THEUPDATE FORMULA REFEQBERLMASS1 WE NOTE THAT LN1  MAXLNNM  LMUSING REFEQLUPDATE2 WE FIND THATLN1  MAXLNN1 LNTHE LFSR BEFORE THE LASTLENGTH CHANGE ALSO SATISFIES BY THE INDUCTIVE HYPOTHESISBEGINEQUATIONYJ  SUMI1LM CIM YJI  BEGINCASES  0  JLM LM1 LDOTS M1 DM  JMENDCASESLABELEQLFSR6ENDEQUATIONTHE OUTPUT OF THE LFSR CN1D WHERE CN1D ISOBTAINED REFEQCUPDATE CAN BE WRITTEN ASYJ  SUMI1LN1 CN1YJI  YJ  SUMI1LN  CNI YJI  DN DM1LEFTYJNM  SUMI1LM CIM YJNMIRIGHTTHEN USING REFEQLFSR5 AND REFEQLFSR6 WE HAVE YJ  SUMI1LN1 CN1YJI  BEGINCASES  0  JLN1 LN11 LDOTS N1 DN  DNDM1DM  0  JNENDCASESSO THAT THE LFSR PRODUCES Y0Y1 LDOTS YNFROM REFEQCUPDATE AND REFEQLUPDATE2 IT FOLLOWS THAT THEDEGREE OF CN1DMUST SATISFY LN1  MAXLNNMLM  MAXLNN1LNENDPROOFIN THE UPDATE STEP WE OBSERVE THAT IF  2LN  NTHEN USING REFEQLUPDATE CN1 HAS LENGTH LN1 LN THAT IS THE POLYNOMIAL IS UPDATED BUT THERE IS NO CHANGE INLENGTH  THE SHIFTREGISTER SYNTHESIS ALGORITHM KNOWN AS MASSEYS ALGORITHMIS PRESENTED FIRST IN PSEUDOCODE AS ALGORITHM REFALGMASSEYP WHEREWE USE THE NOTATIONS CD  CND QQUAD PD  CMDBEGINPROGENVMASSEYS    ALGORITHM PSEUDOCODEMASSEYPMASSEYS ALGORITHM PSEUDOCODESMALLBEGINPROGTABSQUAD  QUAD  QUAD  QUAD QUAD  QUAD  QUAD KILLINPUT  Y0Y1LDOTSYN1 INITIALIZE L  0 CD  1  THE CURRENT CONNECTION POLYNOMIAL PD  1 THE CONNECTION POLYNOMIAL BEFORE LAST LENGTH CHANGE S  1 S IS NM THE AMOUNT OF SHIFT IN UPDATE DM  1 PREVIOUS DISCREPANCY FOR N0 TO N1     D  YN  SUMI1L CI YNI    IF D  0       S  S1  ELSE  IF 2 L  N THEN NOLENGTH CHANGE IN UPDATE     CD  CD D DM1 DS PD     S  S1       ELSE      UPDATE C WITH LENGTH CHANGE    TD  CD TEMPORARY STORE    CD  CD D DM1 DS PD    L  N1L    PD  TD    DM  D    S  1  END ENDENDPROGTABSENDPROGENVA SC MATLAB IMPLEMENTATION OF MASSEYS ALGORITHM WITH COMPUTATIONSOVER GF2 IS SHOWN IN ALGORITHM REFALGMASSEY  THE VECTORIZEDSTRUCTURE OF SC MATLAB ALLOWS THE PSEUDOCODE IMPLEMENTATION TO BEEXPRESSED ALMOST DIRECTLY IN EXECUTABLE CODE  THE STATEMENT TT C   MODC ZEROS1LMSLN  ZEROS1S P2 SIMPLY ALIGNS THEPOLYNOMIALS REPRESENTED IN TT C AND TT P BY APPENDING ANDPREPENDING THE APPROPRIATE NUMBER OF ZEROS AFTER WHICH THEY CAN BEADDED DIRECTLY ADDITION IS MOD 2 SINCE OPERATIONS ARE IN GF2RENEWCOMMANDEXPLICITPROG1RENEWCOMMANDPROGDIRMATLABDIRBEGINNEWPROGENVMASSEYS ALGORITHMMASSEYMMASSEYMASSEYS    ALGORITHMENDNEWPROGENVRENEWCOMMANDEXPLICITPROGRENEWCOMMANDPROGDIRBECAUSE THE SC MATLAB CODE SO CLOSELY FOLLOWS THE PSEUDOCODE ONLYA FEW OF THE ALGORITHMS THROUGHOUT THE BOOK WILL BE SHOWN USINGPSEUDOCODE WITH PREFERENCE GIVEN TO SC MATLAB CODE TO ILLUSTRATEAND DEFINE THE ALGORITHMS  TO CONSERVE PAGE SPACE SUBSEQUENT ALGORITHMS ARE NOT EXPLICITLYDISPLAYED  INSTEAD THE ICON PAR  NOINDENT INCLUDEGRAPHICSPICTUREDIRPICON1PS NOINDENT INCLUDEGRAPHICSPICON EPSFIGFILEPICOM EPSFIGFILEPICTUREDIRPICON1PS NOINDENTIS USED TO INDICATE THAT THE ALGORITHM IS TO BE FOUND ON THE CDROMBEGINEXAMPLEFOR THE SEQUENCE OF EXAMPLE REFEXMMASS1 Y   1110100THE FEEDBACK CONNECTION POLYNOMIAL OBTAINED BY A CALL TO TT MASSEYIS C  1101WHICH CORRESPONDS TO THE POLYNOMIAL CD  1DD3THUS YZ1Z1Z3  0OR YJ  YJ1  YJ3AS EXPECTEDENDEXAMPLEBEGINEXERCISES  ITEM PREPARE A TABLE SHOWING THE STORAGE CONTENTS AND OUTPUTS FOR    THE LFSR SHOWN BELOW WITH INITIAL CONDITIONS SHOWN IN THE DELAY    ELEMENTS  ALSO DETERMINE THE CONNECTION POLYNOMIAL CDBEGINCENTERINPUTPICTUREDIRLFSR5LATEXENDCENTERITEM FOR THE LFSR DESCRIBED BY THE THE POLYNOMIAL 1D D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAMITEM FOR THE INITIAL CONTENTS Y3  1 Y2  0 Y1  0  DETERMINE THE OUTPUT SEQUENCE  HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM FOR THE LFSR DESCRIBED BY THE THE POLYNOMIAL 1D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAMITEM FOR THE INITIAL CONTENTS Y3  1 Y2  0 Y1  0  DETERMINE THE OUTPUT SEQUENCE HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM GIVEN THE SEQUENCE 0001010  BEGINENUMERATE  ITEM DETERMINE THE SHORTESTLENGTH LFSR WHICH COULD PRODUCE THIS    SEQUENCE PERFORMING THE COMPUTATIONS BY HAND  ITEM CHECK YOUR WORK USING ALGORITHM REFALGMASSEY IN SC      MATLAB  ENDENUMERATEITEM WRITE THE OUTPUT SEQUENCE AS A POLYNOMIAL YD  Y0  Y1 D  Y2 D2  CDOTSBEGINENUMERATEITEM ITEM USING REFEQLFSR1 SHOW THAT THE JTH COEFFICIENT IN  YDCD VANISHES FOR JLL1 LDOTS WHERE DEGREECD   L  HENCE WE CAN WRITE CDYD  ZDWHERE ZD  Z0  Z1D   CDOTS  ZL1DL1THUS KNOWING ZD WE CAN FIND THE OUTPUT BY POLYNOMIAL LONG DIVISIONBEGINEQUATIONYD  FRACZDCDLABELEQLFSRDIVENDEQUATIONITEM SHOW THAT THE COEFFICIENTS OF ZD CAN BE RELATED TO THE  INITIAL CONDITIONS OF THE LFSR BY BEGINBMATRIX1 0CDOTS  0  C1  1  CDOTS  0 C2  C1  CDOTS  0 VDOTS CL1  CL2  CDOTS  C1  1 ENDBMATRIXBEGINBMATRIXY0  Y1  Y2  VDOTS  YL1 ENDBMATRIX BEGINBMATRIX Z0  Z1  Z2  VDOTS  ZL1ENDBMATRIXENDENUMERATEITEM LET CD  1D2 WITH INITIAL CONTENTS Y0Y1   10  DETERMINE THE FIRST 6 OUTPUTS USING POLYNOMIAL LONG  DIVISION REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE OBTAINED  DIRECTLY FROM THE LFSRITEM LET CD  1DD2 WITH INITIAL CONTENTS  Y0Y1    11  DETERMINE THE FIRST 6 OUTPUTS USING POLYNOMIAL LONG  DIVISION REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE OBTAINED  DIRECTLY FROM THE LFSRITEM DETERMINE THE SEQUENCE YI OF LENGTH SEVEN GENERATED BY  CD  1DD3 AND CALL ITS LENGTH N  THEN COMPUTE THE CYCLIC  AUTOCORRELATION FUNCTION INDEXAUTOCORRELATION RHOK  FRAC1N SUMI0N1 YI YIKWHERE YIK IS DETERMINED CYCLICLY   PLOT THISAUTOCORRELATION FUNCTIONENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONSOME ASPECTS OF PROOFSLABELSECPROOFSBEGINQUOTESOURCEHPP FERGUSONMATHEMATICS IS SIMPLY SUSTAINED LOGICAL THINKINGENDQUOTESOURCEBEGINQUOTESOURCEPLATOTHERE IS NO ROYAL ROAD TO GEOMETRYENDQUOTESOURCEBEGINQUOTESOURCERICHARD W HAMMINGEM CODING AND INFORMATION     THEORY P 164    SOME PEOPLE BELIEVE THAT A THEOREM IS PROVED WHEN A LOGICALLY  CORRECT PROOF IS GIVEN BUT SOME PEOPLE BELIEVE IT IS PROVED ONLY  WHEN THE STUDENTS SEES WHY IT IS INEVITABLY TRUEENDQUOTESOURCEIN ENGINEERING CLASSES THAT REQUIRE PROOFS IT ALMOST INEVITABLYARISES THAT A STUDENT WILL COMPLAIN THAT HE OR SHE DOES NOT KNOW HOWTO DO PROOFS  THE WAY IT IS USUALLY STATED OF DOING PROOFSSEEMS TO SUGGEST THAT THE STUDENT PERHAPS BELIEVES THERE IS SOMEUNIVERSALLY APPLICABLE METHOD OF DOING PROOFS THAT WILL PROVE ALLPROBLEMS  ON THE ONE HAND THERE IS NO ONE THAT KNOWS HOW TO DOPROOFS OF EVERYTHING  A PROOF REQUIRES INSIGHT UNDERSTANDINGBACKGROUND AND CREATIVITY AND SOME PLAUSIBLE CONJECTURES HAVE THUSFAR ELUDED PROOF AND WILL CONTINUE TO DO SO THAT ITSELF IS ATHEOREM  SOME PROOFS HAVE THE SUBTLETY AND BEAUTY OF A WELLCRAFTEDSONNET  ON THE OTHER HAND MOST PROOFS CONSIST OF CLARIFICATIONS OFPATTERNS THAT HAVE BEEN PREVIOUSLY OBSERVED OR ARE PRECISE STATEMENTSOF SOME FACT  EVERY ENGINEERING STUDENT SHOULD BE ABLE TO DOPROOFS TO SOME EXTENTSIGNAL PROCESSING EMPLOYING MATHEMATICAL CONCEPTS TO ACCOMPLISHENGINEERING PURPOSES OFTEN PRESENTS A DIFFICULT CHALLENGE TOENGINEERING STUDENTS WHO WANT TO KNOW HOW TO USE THE MATERIAL BUTRESIST THE MATHEMATICAL FORMALITIES  IN PARTICULAR THEOREMS ANDPROOFS  NEVERTHELESS THROUGHOUT THIS BOOK MANY OF THE CONCEPTS AREPRESENTED IN A THEOREMPROOF FORMAT AS A MEANS OF ORGANIZATION ANDOPPORTUNITIES FOR PROVING MANY CONCEPTS ARE PROVIDED IN THE EXERCISESTHE FOLLOWING JUSTIFICATIONS ARE PROVIDED FOR REQUIRING PROOFS OFENGINEERING STUDENTSBEGINENUMERATEITEM BECAUSE AN ENGINEER PUTS THINGS TOGETHER WITH AN EYE TO DESIGN  AND UTILITY THE ABILITY TO MOVE FROM A REQUIREMENT SPECIFICATION TO  A FINISHED DESIGN IS AN IMPORTANT SKILL  IN ITS RESTRICTED DOMAIN  PROVING A THEOREM IS NOTHING MORE THAN DESIGN TAKING SPECIFICATIONS  AND USING AVAILABLE COMPONENTS TO PRODUCE A RESULT  THE  SPECIFICATIONS ARE THE HYPOTHESES OF THE THEOREM AND THE AVAILABLE  COMPONENTS ARE WHATEVER KNOWLEDGE CAN BE BROUGHT TO BEAR ON THE  PROBLEM  LIKE MOST DESIGN PROBLEMS THERE MAY BE MANY CORRECT  SOLUTIONS AND MANY INCORRECT APPROACHES  IT IS PERHAPS THE  FLEXIBILITY OF CHOICE EXERCISED AGAINST INFLEXIBLE LOGIC THAT MAKES  PROOFS CHALLENGING  LIKE DESIGN A PROOF MAY REQUIRE TRYING MANY  DIFFERENT AVENUES BEFORE A FRUITFUL APPROACH IS ENCOUNTERED  ITEM A PROOF PROVIDES AN OPPORTUNITY TO REVIEW AND DEEPEN  UNDERSTANDING OF CONCEPTS AND DEFINITIONS THAT HAVE BEEN PRESENTED  TOOLS THAT DONT GET USED OR ARE NOT UNDERSTOOD CORRECTLY WILL NEVER  BECOME USEFUL TOOLS  ITEM AS NEW ALGORITHMS ARE DEVELOPED THEY MUST BE EVALUATED  OFTEN  THIS IS DONE EMPIRICALLY BY MEANS OF COMPUTER SIMULATION OR BY  TESTING OF PROTOTYPES  HOWEVER IT IS BETTER TO HAVE A SENSE OF THE  CORRECTNESS OF A DESIGN BEFORE TOO MANY RESOURCES ARE EXPENDED IN  ITS PROTOTYPING  THE SKILLS DEVELOPED IN LEARNING TO DO PROOFS OF  THEOREMS MAY ASSIST IN EVALUATING AND IMPROVING SIGNAL PROCESSING  ALGORITHMS   ITEM THERE IS NO ESCAPING THE FACT THAT THE SIGNAL PROCESSING  LITERATURE IS VERY MATHEMATICAL  A BROAD MATHEMATICAL VOCABULARY  AND THE ABILITY TO READ MATHEMATICS ARE NECESSARY TO DRAW MEANINGFUL  INFORMATION FROM THE LITERATURE  SHOULD THE OCCASION ARISE WHEN  STUDENTS WISH TO PUBLISH THEIR OWN RESULTS IN SIGNAL PROCESSING  LITERATURE THEY WILL NEED TO SPEAK THE LANGUAGE  ITEM DOING A PROOF IS A GOOD CHANCE TO STRETCH SOME INTELLECTUAL  MUSCLESENDENUMERATETHE INTENT OF THIS SECTION IS TO PROVIDE SOME SUGGESTIONS ON METHODSOF PROOF THAT APPEAR IN THE LITERATURE  THIS IS BY NO MEANS ANEXHAUSTIVE LIST NEW AND IMPORTANT CONCEPTS CAN ARISE AS NEW WAYS OFANSWERING QUESTIONS ARE CREATED  AS AN EXAMPLE CONSIDER SHANNONSINDEXCHANNELCODING THEOREM CHANNELCODING THEOREM WHICH STATESBASICALLY THAT THERE IS A CODE WHICH CAN BE USED TO TRANSMIT DATAOVER A CHANNEL WITH ARBITRARILY LOW PROBABILITY OF ERROR PROVIDEDTHAT THE RATE OF TRANSMISSION IS LESS THAN THE CAPACITY OF THECHANNEL  IN PROVING THE THEOREM SHANNON TOOK AN UNPRECEDENTED STEPINSTEAD OF LOOKING FOR A PARTICULAR CODE TO ANSWER THE QUESTION HEINSTEAD AVERAGED OVER ALL POSSIBLE CODES  THIS PARTICULAR TRICK MADETHE ANALYSIS FALL RIGHT INTO PLACE  SUCH TRICKS OR CREATIVEINSIGHTS CANNOT BE TAUGHT  THERE ARE HOWEVER SOME LOGICALAPPROACHES WHICH CAN BE TAUGHT AND EXERCISEDA THEOREM MAY BE STATED SOMETHING LIKE IF P THEN Q  IN THISP IS CALLED THE EM HYPOTHESIS AND Q IS CALLED THE EM  CONCLUSION  WE SAY THAT P IMPLIES Q AND MAY WRITE PRIGHTARROW Q  INDEXIMPLICATION THE STATEMENT IF P THENQ IS NOT LOGICALLY EQUIVALENT TO SAYING THAT BECAUSE Q OCCURSP MUST ALSO OCCUR  FOR EXAMPLE CONSIDER THE FOLLOWING SYLLOGISMSMALLSKIPINDENT IF A BOOK FALLS ON FRANKS HEAD HIS HEAD WILL HURT INDENT  FRANKS HEAD HURTS SMALLSKIPNOINDENT WE CANNOT CONCLUDE THAT A BOOK HAS FALLEN ON FRANKS HEADHE MAY SIMPLY HAVE A HEADACHE  IN THE IMPLICATION P RIGHTARROW QWE SAY THAT P IS SUFFICIENT FOR Q KNOWLEDGE THAT P OCCURS ISSUFFICIENT TO ESTABLISH THE PRESENCE OF Q  HOWEVER P IS NOTNECESSARY FOR Q Q COULD PERHAPS HAVE HAPPENED ANOTHER WAYINDEXSUFFICIENT INDEXNECESSARYNOTE THAT IF P RIGHTARROW Q AND IF Q IS NOT TRUE THEN P CANNOTBE TRUE  BASED ON THE SYLLOGISM ABOVE IF FRANKS HEAD DOES NOT HURTWE EM CAN CONCLUDE THAT A BOOK DID NOT FALL ON HIS HEADEQUIVALENT WAYS OF EXPRESSING THIS IMPLICATION ARE SMALLSKIPP IMPLIES Q INDENT IF P THEN Q INDENT P RIGHTARROW Q INDENT Q IF P INDENT P ONLY IF Q  INDENT P IS A SUFFICIENT BUT NOT NECESSARY CONDITION FOR Q  INDENT NOT Q IMPLIES NOT P THIS IS THE EM  CONTRAPOSITIVEINDEXCONTRAPOSITIVE  INDENT Q IS A NECESSARY CONDITION FOR P SMALLSKIPFOR THE STATEMENT P RIGHTARROW QTHE STATEMENT OBTAINED BY REVERSING THE ROLES OF P AND Q Q RIGHTARROW PIS CALLED THE EM CONVERSE  INDEXCONVERSE THAT FACT THAT PRIGHTARROW Q AND ITS CONVERSE Q RIGHTARROW P ARE BOTH TRUECAN BE STATED IN A VARIETY OF EQUIVALENT WAYS SMALLSKIPP IMPLIES Q EM AND Q IMPLIES P  INDENT P IMPLIES Q AND CONVERSELY INDENT P IF AND ONLY IF Q INDEXIFFSEEIF AND ONLY IF INDEXIF AND ONLY IF INDENT P IS A NECESSARY AND SUFFICIENT CONDITION FOR Q INDENT P LEFTRIGHTARROW Q SMALLSKIPNOINDENT THE STATEMENT P IF AND ONLY IF Q IS OFTEN ABBREVIATEDP IFF Q SMALLSKIPWE NOW PRESENT SOME COMMENTS ABOUT PROOFS IN A GENERAL FRAMEWORKTHESE SUGGESTIONS DO NOT PROVIDE AN EXHAUSTIVE BAG OF TRICKS BUT AREMERELY INTENDED TO SUGGEST SOME APPROACHES THAT MIGHT WORK SUBSECTIONPROOF BY COMPUTATION DIRECT PROOFLABELSECPROOFCOMPINDEXPROOFBY COMPUTATION PROOFS OF SOME STATEMENTS MAY BEMOSTLY COMPUTATIONAL INVOLVING SUCH TECHNIQUES AS INTEGRATION OFTENUSING CHANGE OF VARIABLES PROPERTIES OF INTEGRATION LINEAR ALGEBRATAYLOR SERIES ETC  AS A SIMPLE EXAMPLE TO PROVE THAT CONVOLUTIONCOMMUTES THAT IS THAT INTINFTYINFTY FTTAUHTAUDTAU INTINFTYINFTY FTAUHTTAUDTAUIT SUFFICES TO MAKE A CHANGE OF VARIABLE XTTAU IN THE FIRSTINTEGRAL  IF YOU WERE APPROACHING THE PROBLEM WITHOUT KNOWING THETRICK THE BEST THING TO DO WOULD BE TO SIMPLY TRY SEVERALAPPROACHES  IF WHAT YOU ARE TRYING TO PROVE IS TRUE SOONER OR LATERYOU MAY STUMBLE ACROSS THE CORRECT APPROACH  WHILE THIS MAY LACKPOLISH IT MIRRORS THE WAY THINGS ARE DISCOVERED IN THE REAL WORLDRARELY DOES A USEFUL CONCEPT OR PRODUCT SPRING FORTH FULLBLOWN AS IFFROM THE HEAD OF ZEUS  DISCOVERY REQUIRES EXPLORATION THOUGHT ANDTRIALANDERROR  OF COURSE EXPERIENCE IN AN AREA CAN SHORTEN THETIME BETWEEN CONCEPT AND EXECUTION  TO EXPERIENCED MATHEMATICIANSSOME THINGS BECOME TRANSPARENTLY OBVIOUS BECAUSE THEY HAVE SOLVED SO MANYRELATED PROBLEMS  A STUDENT STARTING OUT IN AN AREA MAY NOT HAVE THEBENEFIT OF THAT INSIGHT  WHAT IS OFTEN REQUIRED IS THE DETERMINATIONTO TRY THINGS OUT POSSIBLY WITHOUT BEING ABLE TO FORESEE AT THEOUTSET WHAT WILL RESULT  EXPERIENCE WILL LENGTHEN THE NUMBER OF STEPSYOU CAN SEE AHEADBEGINEXAMPLEHERE IS AN EXAMPLE OF A DIRECT PROOF  IT NOT ONLY ILLUSTRATES AUSEFUL PROOF BUT INTRODUCES SEVERAL CONCEPTS WHICH WILL BE MORETHOROUGHLY EXPLORED LATER IN THE BOOK SUCH AS DISTANCE MEASURESTRIANGLE INEQUALITY AND NORMS OF VECTORSLET X  XBF1 XBF2 LDOTS XBFM BE A SET OF DISCRETE POINTSIN RBBN  LET DXBFIXBFJ INDICATE THE DISTANCE BETWEEN THEVECTORS XBFI AND XBFJ  THE SETS DEFINED BY VI   XBF IN RBBNMC  XBF TEXT IS CLOSER TO XBFI THAN TO  ANY OTHER XBFJ I NEQ JTHAT IS  VI   XBF IN RBBNMC   DXBFXBFI  DXBFXBFJ I NEQJARE CALLED THE EM VORONOI REGIONS OF X  THE VECTOR XBFI INVI IS CALLED THE CELL REPRESENTATIVE  INDEXVORONOI REGIONVORONOI REGIONS ARISE IN VECTOR QUANTIZATION INDEXVECTOR QUANTIZATION AND DATA COMPRESSION INDEXDATA COMPRESSIONSEE SECTION REFSECCLUSTAPP  WE WILL PROVE THAT VORONOIREGIONS ARE CONVEX SETS INDEXCONVEX SET  PICK A VORONOI CELLWITHOUT LOSS OF GENERALITY WE WILL CALL THE CELL V1 WITH ITS CELLREPRESENTATIVE XBF1  LET PBF AND QBF BE ARBITRARY POINTS IN V1 AND LET US  DESIGNATE PBF AS THE POINT WHICH IS FURTHER FROM XBF1  IF EVERY POINT  ON THE LINE BETWEEN PBF AND QBF IS IN V1 THEN THE SET IS  CONVEX  LET XBF BE A POINT ON THE LINE BETWEEN PBF AND  QBF XBF  LAMBDA PBF  1LAMBDA QBF QQUAD 0 LEQ LAMBDA LEQ 1WE WILL DENOTE DXBF1XBF AS XBF1  XBF THE NORM OFTHE DIFFERENCE  THEN BEGINALIGNED DXBF1XBF    XBF1  LAMBDA PBF  1LAMBDA QBF    LAMBDAXBF1  PBF  1LAMBDAXBF1  QBF LEQ LAMBDA  XBF1  PBF   1LAMBDA XBF1  QBF LEQ LAMBDA  XBF1  PBF LEQ  XBF1  PBFENDALIGNEDWHERE THE FIRST INEQUALITY FOLLOWS FROM THE TRIANGLE INEQUALITYINDEXINEQUALITIESTRIANGLE  THUS XBF IS CLOSER TO XBF1 THANIS PBF WHICH IS IN THE VORONOI CELL  BY THE DEFINITION OF THEVORONOI CELL IF PBF IS IN THE VORONOI CELL THEN XBF MUST ALSOBEENDEXAMPLEOF COURSE THE TRIALANDERROR ASPECT OF FINDING THE CORRECTCOMPUTATION IN THIS EXAMPLE IS NOT SHOWN ONLY THE FINISHED PRODUCTSOME STANDARD TRICKS THAT ARE EMPLOYED IN PROOFS ARE WORTH MENTIONINGBEGINENUMERATEITEM COUNTING AND LISTS  MAKE AN EXHAUSTIVE LIST OF ALL THE  ELEMENTS AND CONSIDER WHAT YOU ARE TRYING TO DO APPLIED TO ALL OF  THEMITEM TO SHOW THAT A AND B ARE THE SAME IT MAY WORK TO SHOW THAT  A SUBSET B AND B SUBSET A  SIMILARLY TO SHOW THAT XY  SHOW THAT X GEQ Y AND Y GEQ X  SEE FOR EXAMPLE THE PROOF  TO THEOREM REFTHMBASISSAMEITEM IN ANALYTICAL WORK THE TAYLOR SERIES OR THE MEAN VALUE  THEOREM ARE EXCELLENT TOOLSITEM EXHAUSTIVE CHECKING  FOR EXAMPLE TO VERIFY THAT A SET  SATISFIES CERTAIN PROPERTIES SIMPLY VALIDATE THAT THE PROPERTIES  HOLD INDIVIDUALLYENDENUMERATESUBSECTIONPROOF BY CONTRADICTIONLABELSECPROOFCONTBEGINQUOTESOURCEAYN RANDEM ATLAS SHRUGGED  P 188CONTRADICTIONS DO NOT EXIST  WHENEVER YOU THINK THAT YOU ARE FACING ACONTRADICTION CHECK YOUR PREMISES  YOU WILL FIND THAT ONE OF THEM ISWRONG INDEXPROOFBY CONTRADICTIONENDQUOTESOURCEA POWERFUL PROOF TECHNIQUE IS PROOF BY CONTRADICTION  IN ORDER TOSHOW THAT P RIGHTARROW Q WE TAKE AS TRUE THE HYPOTHESIS P ANDEM ASSUME THAT Q IS NOT TRUE  THE PROOF FOLLOWS BY SHOWING THATTHIS ASSUMPTION LEADS TO A LOGICAL CONTRADICTION  BEGINEXAMPLE  WE WILL PROVE A MILLENNIAOLD THEOREM KNOWN TO THE PYTHAGOREANS OF  GREECE  RECALL THAT A RATIONAL NUMBER IS A NUMBER  THAT CAN BE EXPRESSED AS A RATIO OF INTEGERS  THUS 37 IS A  RATIONAL NUMBERNOINDENT BF THEOREM EM SQRT2 IS IRRATIONAL SMALLSKIPINDEXIRRATIONALPRIOR TO ESTABLISHING THIS THEOREM THE PYTHAGOREANS HELD THEVIEWPOINT THAT THE HARMONIES OF COSMOS COULD BE EXPRESSED AS RATIOS OFINTEGERS  THIS THEOREM LEAD TO CONSIDERABLE RELIGIOUS UPHEAVAL IN ITSDAY SMALLSKIP NOINDENT BF PROOF WE WILL ASSUME A RESULT CONTRARY TO THESTATEMENT OF THE THEOREM AND SHOW THAT THIS LEADS TO A CONTRADICTIONWE ASSUME THAT SQRT2 EM IS RATIONAL THAT IS THATBEGINEQUATIONSQRT2  MNLABELEQSQRT2ENDEQUATIONFOR SOME INTEGERS M AND N  NOW WE SHOW THAT THIS LEADS TO ACONTRADICTION  SQUARING REFEQSQRT2 WE OBTAINBEGINEQUATION2  FRACM2N2LABELEQPROOFCONT1ENDEQUATIONSO 2N2  M2FROM THIS WE SEE THAT M2 MUST BE AN EVEN NUMBER AND HENCE THAT MMUST BE EVEN SHOW THIS  LET US WRITE M  2K FOR SOME INTEGERK  SUBSTITUTING THIS INTO REFEQPROOFCONT1 WE OBTAIN 2  FRAC4K2N2OR 2  FRACN2K2THIS IS EQUIVALENT TO SQRT2  FRACNKNOW WE HAVE RETURNED BACK AN EXPRESSION HAVING THE SAME FORM ASREFEQSQRT2 BUT WITH K  N  BEING NOW IN A POSITION TO REPEATTHE OPERATION WE HAVE REACHED THE PRECIPICE LEADING TO ACONTRADICTION BECAUSE THE NUMBERS IN THE RATIO WILL BE REDUCED BYITERATION OF THESE SAME STEPS DOWN TO ABSURDLY SMALL VALUES  BY THISCONTRADICTION WE MUST CONCLUDE THAT THE ORIGINAL ASSUMPTIONREFEQSQRT2 IS FALSEENDEXAMPLE EXAMPLES OF PROOF BY CONTRADICTION ARE GIVEN IN THEOREMS REFONE OF THE ISSUES OVER WHICH MATHEMATICIANS SOMETIMES FRET IS THEUNIQUENESS OF A SOLUTION TO A GIVEN PROBLEM  PROVING UNIQUENESS ISVERY COMMONLY DONE USING CONTRADICTION  TWO DISTINCT SOLUTIONS TO THEPROBLEM ARE PROPOSED AND IT IS SHOWN THAT THESE SOLUTIONS ARE EQUALA CONTRADICTION WHICH POINTS OUT THAT ONLY ONE SOLUTION IS POSSIBLETHIS METHOD IS EXEMPLIFIED IN THE PROOF OF THEOREM REFTHMUNIQBASSUBSECTIONPROOF BY INDUCTIONLABELSECPROOFINDBEGINQUOTESOURCEHENRI POINCAREEM SCIENCE AND HYPOTHESIS  THE ESSENTIAL CHARACTERISTIC OF REASONING BY RECURRENCE IS THAT IT  CONTAINS CONDENSED SO TO SPEAK IN A SINGLE FORMULA AN INFINITE  NUMBER OF SYLLOGISMSENDQUOTESOURCEINDEXPROOFBY INDUCTIONPROOF BY INDUCTION ALLOWS ONE TO ESTABLISH GENERAL CONCLUSIONS FROM ALIMITED SET OF TEST CASES  SUPPOSE YOU HAVE SOME STATEMENT THATDEPENDS UPON AN INTEGER N  WE WILL DENOTE THIS STATEMENT BY SN STATEMENT S IS A FUNCTION OF N  YOU BEGIN BY SHOWING THATSN IS TRUE FOR N1  SOMETIMES ANOTHER SMALL VALUE OF N ISTHE STARTING POINT  THEN YOU SHOW THAT ASSUMING SN IS TRUE LEADSTO AN IMPLICATION THAT SN1 IS ALSO TRUE  WHAT IS AMAZING ANDPOWERFUL IS THAT YOU GET TO EM ASSUME THE TRUTH OF SN AND USETHIS TO SHOW THE TRUTH OF SN1  THE ASSUMED HYPOTHESIS SN ISCALLED THE EM INDUCTIVE HYPOTHESISBEGINEXAMPLE  THE FIRST EXAMPLE SHOULD BE FAMILIAR  WE WANT TO SHOW THAT THE SUM  OF THE FIRST N INTEGERS IS SUMK0N K  FRACNN12CLEARLY THIS IS TRUE FOR N0 AND ALSO CLEARLY IT IS TRUE FOR N1LET US ASSUME ITS TRUTH FOR N  THAT IS WE NOW EM ASSUME THAT SUMK0N K  FRACNN12AND SHOW THAT THIS IMPLIESTHE TRUTH FOR N1  THAT IS WE NEED TO SHOW THAT SUMK0N1   FRACN1N22WE HAVEBEGINALIGNEDSUMK0N1 K  LEFTSUMK0N KRIGHT  N1    FRACNN12  N1  FRACN23N22  FRACN1N22ENDALIGNEDWHERE THE SECOND EQUALITY COMES BY ASSUMPTION OF THE INDUCTIVEHYPOTHESISENDEXAMPLE  WE WILL DO ANOTHER INDUCTIVE PROOF OF MATHEMATICAL FLAVOR TO  ILLUSTRATE ANOTHER POINTBEGINEXAMPLE  WE WILL SHOW THAT TEXTIF  N GEQ 5  TEXT THEN  2N  N2WHAT MAKES THIS EXAMPLE FUNDAMENTALLY DIFFERENT FROM THE PREVIOUS ISTHAT THE STARTING POINT IS NOT N0 BUT N5THE STATEMENT IS CLEARLYTRUE WHEN N5  LET US ASSUME THAT IT HOLDS FOR N THAT IS OURINDUCTIVE HYPOTHESIS IS 2N  N2AND SHOW THAT IT MUST BE TRUE FOR N1 THAT IS 2N1  N12WE HAVE BEGINALIGN2N1  2CDOT 2N NONUMBER  2N2 QQUADTEXTBY THE INDUCTIVE HYPOTHESIS NONUMBER  INTERTEXTHFILLBY THE INDUCTIVE HYPOTHESIS N2N2 GEQ N2  5N QQUADTEXTBECAUSE N GEQ 5 NONUMBER  N2  2N3N  N2  2N1 NONUMBER  N12 NONUMBERENDALIGNENDEXAMPLEWE NOW OFFER AN EXAMPLE WITH A LITTLE MORE OF AN ENGINEERING FLAVORBEGINEXAMPLE  SUPPOSE THERE IS A COMMUNICATION LINK IN WHICH ERRORS CAN BE MADE  WITH PROBABILITY P  THIS LINK IS DIAGRAMMED IN FIGURE  REFFIGBSC1A  WHEN A 0 IS SENT IT IS RECEIVED AS A 0 WITH  PROBABILITY 1P AND AS A 1 WITH PROBABILITY P  THIS  COMMUNICATIONLINK MODEL IS CALLED A BINARY SYMMETRIC CHANNEL  BSC INDEXBINARY SYMMETRIC CHANNEL  NOW SUPPOSE THAT N BSCS ARE PLACED END TO END AS IN FIGURE  REFFIGBSC1B  DENOTE THE PROBABILITY OF ERROR AFTER N  CHANNELS BY PNE  WE WISH TO SHOW THAT THE ENDTOEND  PROBABILITY OF ERROR ISBEGINEQUATION PNE  FRAC12112PNLABELEQPNEENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREA SINGLE CHANNELINPUTPICTUREDIRBSC1 QQUADSUBFIGUREN CHANNELS ENDTOENDINPUTPICTUREDIRBSC3    CAPTIONBINARY SYMMETRIC CHANNEL MODEL    LABELFIGBSC1  ENDCENTERENDFIGUREWHEN N1 WE COMPUTE P1E  P AS EXPECTED  LET US NOW ASSUMETHAT PNE AS GIVEN IN REFEQPNE IS TRUE FOR N AND SHOW THATTHIS PROVIDES A TRUE FORMULA FOR PN1E  IN N1 STAGES WE CAN MAKE AN ERROR IF THERE ARE NO ERRORS IN THEFIRST N STAGES AND AN ERROR OCCURS IN THE LAST STAGE OR IF AN ERRORHAS OCCURRED OVER THE FIRST N STAGES AND NO ERROR OCCURS IN THE LASTSTAGE  THUS BEGINALIGNEDPN1E  1PPNE  P1PNE  1PFRAC12112PN  P1FRAC12112PN  QQUADQQUAD TEXTBY THE INDUCTIVE HYPOTHESIS  FRAC12112PN1ENDALIGNEDENDEXAMPLEPROOF BY INDUCTION IS VERY POWERFUL AND WORKS IN A REMARKABLE NUMBEROF CASES  IT REQUIRES THAT YOU BE ABLE TO STATE THE THEOREM YOU MUSTSTART WITH THE INDUCTIVE HYPOTHESIS WHICH IS USUALLY THE DIFFICULTPART  IN PRACTICE STATEMENT OF THE THEOREM MUST COME BY SOME INITIALGRIND SOME INSIGHT AND A LOT OF WORK  THEN INDUCTION IS USED TOPROVE THAT THE RESULT IS CORRECT  SOME SIMPLE OPPORTUNITIES FORSTATING THE INDUCTIVE HYPOTHESIS AND THEN PROVING IT ARE PROVIDED INTHE EXERCISESBEGINEXERCISESITEM SHOW THAT SQRT3 IS IRRATIONALITEM SHOW THAT THERE ARE AN INFINITE NUMBER OF  PRIMES  HINT  USE A PROOF BY CONTRADICTION ASSUMING THAT THERE  ARE ONLY A FINITE NUMBER OF PRIMES  THEN BUILD A NUMBER 2CDOT 3  CDOT 5 CDOT CDOTS CDOT P  1 WHERE P IS THE ASSUMED LAST  PRIME AND SHOW THAT THIS IS NOT DIVISIBLE BY ANY OF THE LISTED PRIMESITEM USING PROOF BY CONTRADICTION SHOW THAT SQRT2 CANNOT BE A  RATIONAL NUMBER  HINT  ASSUME SQRT2  MN FOR SOME INTEGER  M AND N WHERE THE FRACTION IS EXPRESSED IN REDUCED FORM  SHOW  THAT THIS LEADS TO A CONTRADICTIONITEM SHOW THAT IF M2 IS EVEN THEN M MUST BE EVENITEM BY TRIAL AND ERROR DETERMINE A PLAUSIBLE FORMULA FOR SUMI0N 2ITHEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR THE SUM OF THE  FIRST N ODD INTEGERS 135 CDOTS  2N1THEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR SUMI1N FRAC1I2 1THEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM SHOW BY INDUCTION FOR EVERY POSITIVE INTEGER N THAT N3  N  IS IS DIVISIBLE BY 3ITEM THE QUANTITY BOXED NCHOOSEK  FRACNKNK IS THE NUMBER OF WAYS OF CHOOSING K OBJECTS OUT OF N OBJECTSWHERE N GEQ K  THE QUANTITY NCHOOSE K IS ALSO KNOWN AS THE EMBINOMIAL COEFFICIENT  WE READ THE NOTATION N CHOOSE K AS NCHOOSE K INDEXBINOMIAL COEFFICIENT INDEXNKN CHOOSE K  SHOW BY INDUCTION THAT N1CHOOSEK  NCHOOSEK  NCHOOSEK1ITEM SHOW BY INDUCTION THAT FOR N GEQ 0 SUMI0N N CHOOSE K  2NITEM SHOW BY INDUCTION THAT BOXEDXYN  SUMK0N N CHOOSE K XK YNKTHIS IMPORTANT FORMULA IS KNOWN AS THE EM BINOMIAL  THEOREM INDEXBINOMIAL THEOREMITEM PROVE THE FOLLOWING BY INDUCTION SUMK1N J2  FRACNN12N16ITEM PROVE THE FOLLOWING BY INDUCTION BOXED SUMK1N RN  FRACRN  1R1 QQUAD R NEQ 1 INDEXGEOMETRIC SUMITEM PROVE BY INDUCTION THAT FRAC1SQRT4N1  FRAC12CDOT FRAC34 CDOT CDOTSCDOT FRAC2N32N2CDOT FRAC2N12N   FRAC1SQRT3N1FOR INTEGERS N GEQ 2KAZARINOFF 1961 P 5ITEM PROVE BY INDUCTION THAT FOR XYN IN ZBB XY DIVIDES  INDEX INDEXDIVIDESSEE  XNYN  THIS IS WRITTEN AS XY  XNYNENDEXERCISESINPUTLINALGDIRLININTRO  INTRODUCT TO LINEAR ALGEBRA CHAPTER VECTOR SPACESINPUTLINALGDIRVECTSP    VECTOR SPACES FINITE DIMENSIONAL ANDINPUTLINALGDIRVECT    VECTOR SPACES APPLICATIONS CHAPTER MATRICES AND LINEAR OPERATORSINPUTLINALGDIRMATEQ     MATRIX EQUATIONS   INCLUDES NORM AND RANK CHAPTER COMPUTING MATRIX SOLUTIONSINPUTLINALGDIRMATINV    PROPERTIES OF MATRIX INVERSESINPUTLINALGDIRMATCOND   MATRIX CONDITION NUMBER LU FACTORIZATIONINPUTLINALGDIRMATPROJ   PROJECTION MATRICESINPUTLINALGDIRLINTRANS  TRANSFORMATION OF BASES SIMILARITY INPUTLINALGDIREIGEN     STUFF ON EIGENVECTORSINPUTLINALGDIRMATFACT    MATRIX FACTORIZATIONSINPUTLINALGDIRSVD       SINGULAR VALUE DECOMPOSITIONINPUTLINALGDIRCANON     CANONICAL FORMS FOR MATRICESINPUTLINALGDIRMODALMATRIX  MODAL MATRICES AND EXPONENTIAL MODELSINPUTLINALGDIRSPECIALMAT  SPECIAL MATRIX FORMSINPUTLINALGDIRKRONECKER  KRONECKER PRODUCT AND ITS APPLICATIONSINPUTLINALGDIRVECOPAPPENDIXINPUTLINALGDIRLINBASICSMYENDAPPENDIX LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERSIGNAL SPACESLABELCHAPVECTSPBEGINQUOTESOURCEEDWARD ABBEYEM DESERT SOLITAIRE  LANGUAGE MAKES A MIGHTY LOOSE NET WITH WHICH TO GO FISHING FOR  SIMPLE FACTS WHEN FACTS ARE INFINITEENDQUOTESOURCEBEGINQUOTESOURCEHENRI POINCAREEM SCIENCE AND HYPOTHESIS  BEGINNERS ARE NOT PREPARED FOR REAL MATHEMATICAL RIGOR THEY WOULD  SEE IN IT NOTHING BUT EMPTY TEDIOUS SUBTLETIES  IT WOULD BE A  WASTE OF TIME TO TRY TO MAKE THEM MORE EXACTING THEY HAVE TO PASS  RAPIDLY AND WITHOUT STOPPING OVER THE ROAD WHICH WAS TRODDEN SLOWLY  BY THE FOUNDERS OF THE SCIENCEENDQUOTESOURCEBEGINQUOTESOURCECHAIM POTOKEM IN THE BEGINNING  ALL BEGINNINGS ARE HARDENDQUOTESOURCETHIS CHAPTER IS MOSTLY ABOUT TWO KINDS OF MATHEMATICAL OBJECTS METRICSPACES AND LINEAR VECTOR SPACES  THE IDEA BEHIND A METRIC SPACE ISSIMPLY THAT WE PROVIDE A WAY OF MEASURING THE DISTANCE BETWEENMATHEMATICAL OBJECTS SUCH AS SETS POINTS FUNCTIONS OR SEQUENCESWITH THIS NOTION OF DISTANCE WE WILL BE ABLE TO GENERALIZE SOME OF THEFAMILIAR CONCEPTS OF CALCULUS SUCH AS CONTINUITY OR CONVERGENCEBEYOND OPERATIONS ON A SINGLE DIMENSION TO OPERATIONS IN HIGHERDIMENSIONS  THE CONCEPT OF A VECTOR SPACE IS ALSO SIMPLE IT IS A SET OF OBJECTSTHAT CAN BE COMBINED TOGETHER USING LINEAR COMBINATIONS  BUT THETHEORY OF VECTOR SPACES HAS FARREACHING RAMIFICATIONS COVERING ASIGNIFICANT PORTION OF THE THEORY OF SIGNAL PROCESSING  A KEY INSIGHTIN VECTOR SPACE THEORY IS THAT IN A GEOMETRICALLY USEFUL SENSE BF  FUNCTIONS IE SIGNALS CAN BE REGARDED AS VECTORS  THISGEOMETRIC UNDERSTANDING PROVIDES A POWERFUL TOOL FOR SIGNAL ANALYSISIN THIS CHAPTER THE BASIC THEORY AND NOTATION OF VECTOR SPACES ISDEVELOPED  IN CHAPTER REFCHAPVECTAP WE PUT THIS NOTION TO WORK INA VARIETY OF APPLICATIONS INCLUDING OPTIMAL FILTERING BOTH LEASTSQUARES AND MINIMUM MEAN SQUARES TRANSFORMS DATA COMPRESSIONSAMPLING AND INTERPOLATIONIN OUR STUDY OF METRIC SPACES AND VECTOR SPACES THE INTENT IS TOPROVIDE A FRAMEWORK FOR THE GENERAL DISCUSSION OF SIGNALS  BEFOREEMBARKING ON THIS CHAPTER THE READER IS ENCOURAGED TO REVIEW THEBASIC DEFINITIONS OF FUNCTIONS AND SETS APPEARING IN APPENDIXREFAPPDXSETFUNCT  MATRIX NOTATION IS HEAVILY EMPLOYED IN THISSTUDY SO REVIEW OF THE BASIC MATRIX NOTATIONS PRESENTED IN APPENDIXREFAPPDXLINBASICS IS ALSO RECOMMENDEDIN THE DEVELOPMENT OF THIS CHAPTER WE BUILD SUCCESSIVELY FROM BF METRICSPACES TO BF VECTOR SPACES TO BF NORMED VECTOR SPACES TOBF NORMED INNERPRODUCT SPACES  THIS WILL LEAD US TO THE IMPORTANT IDEA OFPROJECTIONS AND ORTHOGONAL PROJECTIONS  ORTHOGONAL PROJECTION WILL BEA TOOL OF TREMENDOUS IMPORTANCE TO US IN THE NEXT CHAPTER WHERE ITWILL BE USED AS THE GEOMETRICAL BASIS FOR BOTH LEASTSQUARES ANDMINIMUM MEANSQUARES FILTERING AND PREDICTIONINPUTHOMEDIRLINALGPARTBASICSSECTIONSOME ALGEBRAIC DEFINITIONSTHE DEFINITIONS IN THIS SECTION ARE PROVIDED TO BE ABLE TO STATECLEARLY IN WHAT FOLLOWS WHERE THE COMPUTATIONS TAKE PLACE  IN SOMEAPPLICATIONS COMPUTATIONS ARE NOT DONE USING THE FAMILIAR REALNUMBERS BUT ARE DONE USING NUMBERS MODULO N SUCH AS N256 FOR8BIT REPRESENTATIONS  IN THIS CASE WE MUST PAY ATTENTION TO THEPARTICULAR ALGEBRAIC PROPERTIES OF THE OBJECTS THAT ARE USED  THEALGEBRAIC PROPERTIES OF INTEREST ARE WHETHER THE COMPUTATIONS TAKEPLACE IN A GROUP A RING OR A FIELD  IF THE PARTICULAR APPLICATIONSOF INTEREST TO THE READER WILL ALWAYS BE COMPUTED USING REAL NUMBERSRBB THEN THESE DEFINITIONS CAN BE SKIPPED SINCE THE SET OF REALNUMBERS IS A GROUP UNDER BOTH ADDITION AND MULTIPLICATION AFTERREMOVING 0  IT IS ALSO A RING AND A FIELDTO INTRODUCE GROUPS RINGS AND FIELDS WE NEED THE NOTION OF A BINARYOPERATOR  IN THE INTEREST OF BREVITY WE INTRODUCE THIS BY SEVERALEXAMPLESBEGINEXAMPLE   BEGINENUMERATE  ITEM THE OPERATOR  IS A BINARY OPERATOR  ITEM THE OPERATOR  IS A BINARY OPERATOR  ITEM THE OPERATOR CDOT MULTIPLICATION IS A BINARY OPERATOR  ITEM THE FUNCTION COMPOSITION OPERATOR CIRC IS A BINARY OPERATOR  ENDENUMERATEENDEXAMPLEIN SHORT A BINARY OPERATOR TAKES TWO OPERANDS AND RETURNS THEOPERATION ON THOSE TWO OPERANDSSUBSECTIONGROUPSLABELSECGROUPSBEGINDEFINITION LABELDEFGROUP  A SET S EQUIPPED WITH SINGLE BINARY OPERATION  IS A GROUP IF IT  SATISFIES THE FOLLOWINGBEGINENUMERATEG1ITEM THE BINARY OPERATION IS CLOSED IN S  THIS IS DIFFERENT THEN  THE TOPOLOGICAL NOTION OF CLOSURE  THAT IS FOR ANY A B IN S  THE ELEMENTS AB AND BA ARE ALSO IN S INDEXCLOSEDOPERATIONITEM THERE IS AN IDENTITY ELEMENT EIN S SUCH THAT FOR ANY AIN  S  INDEXIDENTITY ELEMENT AE  EA  ATHAT IS THE IDENTITY ELEMENT LEAVES EVERY ELEMENT UNCHANGED UNDER THEOPERATION ITEM FOR EVERY ELEMENT A IN S THERE IS AN ELEMENT B IN S CALLED  ITS EM INVERSE SUCH THAT AB  E QQUAD BA  EITEM THE BINARY OPERATION IS ASSOCIATIVE INDEXASSOCIATIVE  FOR  EVERY AB C IN S ABC  ABCENDENUMERATEWE DENOTE THE GROUP BY LA SRAENDDEFINITIONIF IT IS TRUE THAT AB  BA FOR EVERY AB IN S THEN THE GROUPIS SAID TO BE A BF COMMUTATIVE  INDEXCOMMUTATIVEINDEXABELIAN IF THE OPERATION  IS AN ADDITION OPERATOR ACOMMUTATIVE GROUP IS REFERRED TO AS AN BF ABELIAN GROUP  NOTE THATIT IS NOT NECESSARY FOR EVERY GROUP TO BE COMMUTATIVEEXAMPLES OF GROUPSBEGINENUMERATEITEM THE INTEGERS UNDER ADDITION  THE GROUP IS DENOTED LA  ZBBRA  NOTE THAT THE INTEGERS UNDER MULTIPLICATION DO EM  NOT FORM A GROUP THERE IS NO MULTIPLICATIVE INVERSE FOR EVERY ELEMENTITEM THE INTEGERS MODULO 7 UNDER ADDITION  THIS GROUP IS DENOTED LA  ZBB7RA  ALSO THE INTEGERS MODULO 7 UNDER MULTIPLICATION  DENOTED AS LA ZBB7CDOTRA  HOWEVER INTEGERS MODULO 6 UNDER  MULTIPLICATION DO NOT FORM A GROUP THERE IS NO MULTIPLICATIVE  INVERSE TO THE NUMBER 2 FOR EXAMPLEITEM THE SET OF REAL NUMBERS UNDER EITHER ADDITION OR MULTIPLICATION  LA RBBRA OR LA RBBCDOT RAITEM THE SET OF POLYNOMIALS WITH COEFFICIENTS FROM A GROUPENDENUMERATESUBSECTIONRINGSLABELSECRINGSBEGINDEFINITION LABELDEFRINGINDEXRING A SET R EQUIPPED WITH TWO OPERATIONS WHICH WE WILLDENOTE AS  AND  IS A RING IF IT SATISFIES THE FOLLOWINGBEGINENUMERATER1ITEM LA RRA IS AN ABELIAN GROUPITEM THE OPERATION  IS ASSOCIATIVEITEM LEFT AND RIGHT DISTRIBUTED LAWS HOLD  FOR ALL ABC IN R ABC  ABAC QQUADQQUAD ABC  ACBCENDENUMERATEWE DENOTE THE RING BY LARRAENDDEFINITIONWE NOTE IN PARTICULAR THAT EM MULTIPLICATIVE INVERSES NEED NOT EXIST  IN A RING  IN FACT THE RING MIGHT NOT EVEN HAVE A MULTIPLICATIVE  IDENTITY  THE OPERATOR  IS NOT NECESSARILY COMMUTATIVE NOR IS AN IDENTITY ORINVERSE REQUIRED FOR THE OPERATION   IF THERE IS AN ELEMENT 1 IN R SUCH THAT FOR ANY R IN R 1R  R1  RTHIS ELEMENT IS SAID TO BE AN IDENTITY AND THE RING IS SAID TO BE A RINGWITH IDENTITYEXAMPLES OF RINGSBEGINENUMERATEITEM THE SET OF SQUARE MATRICES WITH REAL ELEMENTS NOT COMMUTATIVE  MULTIPLICATION HAS AN IDENTITY NOT EVERY MATRIX HAS AN INVERSEITEM THE SET OF RATIONAL NUMBERS QBBITEM THE SET OF REAL NUMBERS RBBITEM THE SET OF COMPLEX NUMBERS CBBITEM THE SET OF POLYNOMIALS WHOSE COEFFICIENTS COME FROM A RING  COMMUTATIVE HAS AN IDENTITY POLYNOMIALS MAY NOT HAVE  MULTIPLICATIVE INVERSESITEM THE SET OF POLYNOMIALS WITH MULTIPLICATION DONE MODULO ANOTHER  POLYNOMIALITEM INTEGERS MODULO 6 LA ZBB6CDOTRA  NOT EVERY ELEMENT  HAS A MULTIPLICATIVE INVERSEENDENUMERATESUBSECTIONFIELDSLABELSECFIELDS FIELDS INCORPORATE THE ALGEBRAIC OPERATIONS WE ARE FAMILIAR WITHFROM WORKING WITH REAL AND COMPLEX NUMBERSBEGINDEFINITION LABELDEFFIELD  A F EQUIPPED WITH TWO OPERATIONS  AND  IS A FIELD IF IT  SATISFIES THE FOLLOWINGBEGINENUMERATEF1ITEM LA FRA IS AN ABELIAN GROUPITEM THE SET F EXCLUDING 0 THE ADDITIVE IDENTITY IS A COMMUTATIVE  GROUP UNDER ITEM THE OPERATIONS  AND  DISTRIBUTEENDENUMERATEWE MAY DENOTE THE FIELDS AS LA FCDOTRAENDDEFINITIONEXAMPLES OF FIELDSBEGINENUMERATEITEM THE FAMILIAR OPERATIONS ON THE RATIONALS REALS AND COMPLEX  NUMBERS ITEM THE INTEGERS MODULO 2  THIS FORMS A FIELD THAT ARISES  FREQUENTLY IN DIGITAL OPERATIONS SINCE THE ELEMENTS ARE EITHER 0 OR  1  THIS FIELD IS REFERRED TO AS GF2  INDEXGF2GF2ITEM THE INTEGERS MODULO 7 LA ZBB7CDOTRA  THIS FORMS A  EM FINITE FIELD IT TURNS OUT WE WONT SHOW THIS HERE THAT  FIELD HAVING A FINITE NUMBER OF ELEMENTS HAS PM ELEMENTS IN IT  WHERE P IS PRIME  HOWEVER IF M1 THE OPERATIONS ARE NOT DONE  SIMPLY USING OPERATIONS MODULO PMITEM AS AN EXAMPLE OF SOMETHING EM NOT A FIELD INTEGER OPERATIONS  MODULO 4 DOES NOT FORM A FIELDENDENUMERATESECTIONVECTOR SPACESLABELSECVS1A FINITEDIMENSIONAL VECTOR XBF MAY BE WRITTEN AS XBF  LEFTBEGINARRAYC X1X2 VDOTS XNENDARRAYRIGHTTHE ELEMENTS OF THE VECTOR ARE XI I12LDOTSN  EACH OF THEELEMENTS OF THE VECTOR LIES IN SOME SET SUCH AS THE SET OF REALNUMBERS XI IN RBB OR THE SET OF INTEGERS XI IN ZBB   THISSET OF NUMBERS IS CALLED THE SET OF SCALARS OF THE VECTOR SPACETHE FINITEDIMENSIONAL VECTOR REPRESENTATION IS WIDELY USEDESPECIALLY FOR DISCRETETIME SIGNALS IN WHICH THE DISCRETETIMESIGNAL COMPONENTS FORM ELEMENTS IN A VECTOR  HOWEVER FORREPRESENTING AND ANALYZING CONTINUOUSTIME SIGNALS A MOREENCOMPASSING UNDERSTANDING OF VECTOR CONCEPTS IS USEFUL  IT ISPOSSIBLE TO REGARD THE FUNCTION XT AS A VECTOR AND TO APPLY MANYOF THE SAME TOOLS TO THE ANALYSIS OF XT THAT MIGHT BE APPLIED TOTHE ANALYSIS OF A MORE CONVENTIONAL VECTOR XBF  WE WILL THEREFOREUSE THE SYMBOL X OR XT ALSO TO REPRESENT VECTORS AS WELL ASTHE SYMBOL XBF PREFERRING THE SYMBOL XBF FOR THE CASE OFFINITEDIMENSIONAL VECTORS  ALSO IN INTRODUCING NEW VECTOR SPACECONCEPTS VECTORS ARE INDICATED IN BOLD FONT TO DISTINGUISH THEVECTORS FROM THE SCALARS  NOTE IN HANDWRITTEN NOTATION SUCH AS ON ABLACKBOARD THE BOLD FONT IS USUALLY DENOTED IN THE SIGNAL PROCESSINGCOMMUNITY BY AN UNDERSCORE AS IN XUL OR FOR BREVITY BY NOADDITIONAL NOTATION  DENOTING HANDWRITTEN VECTORS WITH ASUPERSCRIPTED ARROW VECX IS MORE COMMON IN THE PHYSICS COMMUNITYBEGINDEFINITION  A BF LINEAR VECTOR SPACE INDEXVECTOR SPACE S OVER A SET OF  SCALARS R IS A COLLECTION OF OBJECTS KNOWN AS VECTORS TOGETHER  WITH AN ADDITIVE OPERATION  AND A SCALAR MULTIPLICATION OPERATION  CDOT THAT SATISFY THE FOLLOWING PROPERTIESBEGINENUMERATEVS1ITEM S FORMS A GROUP INDEXGROUP UNDER ADDITION   THAT IS  THE FOLLOWING PROPERTIES ARE SATISFIED  BEGINENUMERATE  ITEM FOR ANY XBF AND YBF IN S XBF  YBF IN S  THE    ADDITION OPERATION IS CLOSEDFOOTNOTEA CLOSED OPERATION IS A      DISTINCT CONCEPT FROM A CLOSED SET  A CLOSED BINARY OPERATION      INDEXCLOSED OPERATION  ON A SET S IS SUCH THAT FOR ANY      X Y IN S THEN XY IN SITEM THERE IS AN IDENTITY ELEMENT IN S WHICH WE WILL DENOTE AS  ZEROBF SUCH THAT FOR ANY XBF IN S XBF  ZEROBF  ZEROBF  XBF  XBFITEM FOR EVERY ELEMENT XBF IN S THERE IS ANOTHER ELEMENT YBF  IN S SUCH THAT XBF  YBF  ZEROBFTHE ELEMENT YBF IS THE ADDITIVE INVERSE OF XBF AND IS USUALLYDENOTED AS XBFITEM THE ADDITION OPERATION IS ASSOCIATIVE FOR ANY XBF YBF AND  ZBF IN S XBFYBF ZBF  XBF  YBFZBF  ENDENUMERATEITEM FOR ANY A B IN R AND ANY XBF AND YBF IN S  AXBF IN S  ABXBF  ABXBF  ABXBF  A XBF  BXBF  AXBF  YBF  A XBF  A YBFITEM THERE IS A MULTIPLICATIVE IDENTITY ELEMENT 1 IN R SUCH THAT  1XBF  XBF  THERE IS AN ELEMENT 0 IN R SUCH THAT 0XBF  0ENDENUMERATETHE SET R IS THE SET OF SCALARS OF THE VECTOR SPACEENDDEFINITIONTHE SET OF SCALARS IS MOST FREQUENTLY TAKEN TO BE THE SET OF REALNUMBERS OR COMPLEX NUMBERS  HOWEVER IN SOME APPLICATIONS OTHER SETSOF SCALARS ARE USED SUCH AS POLYNOMIALS OR NUMBERS MODULO 256  THEONLY REQUIREMENT ON THE SET OF SCALARS IS THAT THE OPERATIONS OFADDITION AND MULTIPLICATION CAN BE USED AS USUAL ALTHOUGH NOMULTIPLICATIVE INVERSE IS NEEDED AND THAT THERE IS A NUMBER 1 THATIS A MULTIPLICATIVE IDENTITY  IN THIS CHAPTER WHEN WE TALK ABOUTISSUES SUCH AS CLOSED SUBSPACES COMPLETE SUBSPACES AND SO FORTH ITIS ASSUMED THAT THE SET OF SCALARS IS EITHER THE REAL NUMBERS RBBOR THE COMPLEX NUMBERS CBB SINCE THESE ARE COMPLETEWE WILL REFER INTERCHANGEABLY TO EM LINEAR VECTOR SPACE OR EM  VECTOR SPACEBEGINEXAMPLE THE MOST FAMILIAR VECTOR SPACE IS RBBN THE SET OF  NTUPLES INDEXRNRBBN FOR EXAMPLE  IF XBF1 XBF2 IN RBB4 AND XBF1  BEGINBMATRIX 1  5  4  2 ENDBMATRIXQQUADQQUADXBF2  BEGINBMATRIX 5  2   0  2 ENDBMATRIXTHEN XBF1  XBF2  BEGINBMATRIX6  7  4  0 ENDBMATRIXQQUADQQUAD3 XBF1  2 XBF2  BEGINBMATRIX1319122 ENDBMATRIXENDEXAMPLESEVERAL OTHER FINITEDIMENSIONAL VECTOR SPACES EXIST OF WHICH WEMENTION A FEWBEGINEXAMPLE   BEGINENUMERATE  ITEM THE SET OF MATSIZEMN MATRICES WITH REAL ELEMENTS  ITEM THE SET OF POLYNOMIALS OF DEGREE UP TO N WITH REAL COEFFICIENTS  ITEM THE SET OF POLYNOMIALS WITH REAL COEFFICIENTS WITH THE USUAL    ADDITION AND MULTIPLICATION MODULO THE POLYNOMIAL PT  1T8    FORMS A LINEAR VECTOR SPACE  WE DENOTE THIS VECTOR SPACE AS    RBBTT81  ENDENUMERATEENDEXAMPLEIN ADDITION TO THESE EXAMPLES WHICH WILL BE SHOWN SUBSEQUENTLY TOHAVE FINITE DIMENSIONALITY THERE ARE MANY IMPORTANT VECTOR SPACESTHAT ARE INFINITEDIMENSIONAL IN A MANNER TO BE MADE PRECISE BELOWBEGINEXAMPLE   BEGINENUMERATE  ITEM LP THE SET OF ALL INFINITELYLONG SEQUENCES XN FORMS AN    INFINITEDIMENSIONAL VECTOR SPACE  ITEM CAB THE SET OF CONTINUOUS FUNCTIONS DEFINED    OVER THE INTERVAL AB FORMS A VECTOR SPACE  INDEXCABCAB  ITEM LPAB THE FUNCTIONS IN LP FORM THE ELEMENTS OF AN    INFINITEDIMENSIONAL VECTOR SPACEENDENUMERATEENDEXAMPLEBEGINDEFINITION  LET S BE A VECTOR SPACE  IF V SUBSET S IS A SUBSET SUCH THAT  V IS ITSELF A VECTOR SPACE THEN V IS SAID TO BE A BF    SUBSPACE OF SENDDEFINITIONBEGINEXAMPLE   BEGINENUMERATE  ITEM LET S BE THE SET OF ALL POLYNOMIALS AND LET V BE THE SET    OF POLYNOMIALS OF DEGREE LESS THAN 6  THE V IS A SUBSPACE OF    S  ITEM LET S CONSIST OF THE SET OF 5TUPLES S   00000010011000111000AND LET V BE THE SET V   0000001001 WHERE THE ADDITION IS DONE MODULO 2  THEN S IS A VECTOR SPACECHECK THIS AND V IS A SUBSPACE  ENDENUMERATEENDEXAMPLETHROUGHOUT THIS CHAPTER AND THE REMAINDER OF THE BOOK WE WILL USEINTERCHANGEABLY THE WORDS VECTOR AND SIGNAL  FOR ADISCRETETIME SIGNAL WE MAY THINK OF THE VECTOR COMPOSED OF THESAMPLES OF THE FUNCTION AS A VECTOR IN RBBN OR CBBN  FOR ACONTINUOUSTIME SIGNAL ST THE VECTOR IS THE SIGNAL ITSELF ANELEMENT OF A SPACE SUCH AS LPAB  THUS BOXEDTEXTTHE STUDY OF SIGNALS IS THE STUDY OF VECTOR SPACESBEGINEXERCISES  ITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR INTERSECTION V CAP W IS A SUBSPACEITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR SUM VW IS A SUBSPACEENDEXERCISESSUBSECTIONLINEAR COMBINATIONS OF VECTORSLET S BE A VECTOR SPACE OVER R AND LETPBF1PBF2LDOTSPBFM BE VECTORS IN S THEN FOR CI IN RTHE LINEAR COMBINATION XBF  C1PBF1  C2 PBF2  LDOTS CM PBFM IS IN S  THE SET OF VECTORS PBFI CAN BE REGARDED AS EM  BUILDING BLOCKS OR INGREDIENTS FOR OTHER SIGNALS AND THE LINEARCOMBINATION SYNTHESIZES XBF FROM THESE COMPONENTS  IF THE SET OFINGREDIENTS IS SUFFICIENTLY RICH THAN A WIDE VARIETY OF SIGNALSVECTORS CAN BE CONSTRUCTED  IF THE INGREDIENT VECTORS ARE KNOWNTHEN THE VECTOR XBF IS ENTIRELY CHARACTERIZED BY THE REPRESENTATIONC1C2LDOTSCM SINCE KNOWING THESE TELLS HOW TO SYNTHESIZEXBF  BEGINDEFINITION LABELDEFLC  LET S BE A VECTOR SPACE OVER R AND LET T SUBSET S PERHAPS  WITH INFINITELY MANY ELEMENTS  A POINT XBF IN S IS SAID TO BE  A BF LINEAR COMBINATION INDEXLINEAR COMBINATION OF POINTS IN  T IF THERE IS A EM FINITE SET OF POINTS PBF1PBF2LDOTS  PBFM IN T AND A FINITE SET OF SCALARS C1C2LDOTS CM IN  R SUCH THAT XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFMENDDEFINITIONIT IS SIGNIFICANT THAT THE LINEAR COMBINATION ENTAILS ONLY A FINITESUMBEGINEXAMPLE LABELEXMLINCOMB1LET S  CRBB THE SET OF CONTINUOUS FUNCTIONS  DEFINED ON THE REAL NUMBERS  LET P1T  1 P2T  T AND  P3T  T2  THEN A LINEAR COMBINATION OF THESE FUNCTIONS IS XT  C1  C2T  C3 T2 THESE FUNCTIONS CAN BE USED AS BUILDING BLOCKS TO CREATE ANYSECONDDEGREE POLYNOMIAL  AS WILL BE SEEN IN THE FOLLOWING THEREARE FUNCTIONS BETTER SUITED TO THE TASK OF BUILDING POLYNOMIALSIF THE FUNCTION P4T  T2  1 IS ADDED TO THE SET OF FUNCTIONSTHEN OTHER FUNCTIONS OF THE FORM XT  C1  C2 T  C3T2  C4 T21  C1C4  A2 T C3C4 T2CAN BE CONSTRUCTED WHICH IS STILL JUST A QUADRATIC POLYNOMIAL  THATIS THE NEW FUNCTION DOES NOT EXPAND THE SET OF FUNCTIONS THAT CAN BECONSTRUCTED SO P4T IS IN SOME SENSE REDUNDANT  THIS MEANSTHAT THERE IS MORE THAN ONE WAY TO REPRESENT A POLYNOMIAL  FOREXAMPLE THE POLYNOMIAL XT  6  5T  T2CAN BE REPRESENTED AS XT  8P1T  5 P2T   P3T  2P4T OR AS XT  9P1T  5 P2T   2P3T  3P4T ENDEXAMPLEBEGINEXAMPLELET PBF1PBF2 IN RBB3 WITH PBF1  101T  PBF2  110T  THEN XBF  C1XBF1  C2 XBF2  LEFTBEGINARRAYCC1C2  C2 1ENDARRAYRIGHT THE SET OF VECTORS THAT CAN BE CONSTRUCTED WITH PBF1PBF2DOES NOT COVER THE SET OF ALL VECTORS IN RBB3  FOR EXAMPLE THEVECTOR XBF  BEGINBMATRIX 5  2  6 ENDBMATRIXCANNOT BE FORMED AS A LINEAR COMBINATION OF PBF1 AND PBF2ENDEXAMPLESEVERAL QUESTIONS RELATED TO LINEAR COMBINATIONS ARE ADDRESSED IN THISAND SUCCEEDING SECTIONS AMONG THEMBEGINITEMIZEITEM IS THE REPRESENTATION OF A VECTOR AS A LINEAR COMBINATION OF  OTHER VECTORS UNIQUEITEM WHAT IS THE SMALLEST SET OF VECTORS THAT CAN BE USED TO  SYNTHESIZE ANY VECTOR IN SITEM GIVEN THE SET OF VECTORS PBF1PBF2LDOTSPBFM HOW ARE  THE COEFFICIENTS C1C2LDOTSCM FOUND TO REPRESENT THE VECTOR  XBF IF IN FACT IT CAN BE REPRESENTEDITEM WHAT ARE THE REQUIREMENTS ON THE VECTORS PBFI IN  ORDER TO BE ABLE TO SYNTHESIZE ANY VECTOR X IN SITEM SUPPOSE THAT XBF CANNOT BE REPRESENTED EXACTLY USING THE SET OF  VECTORS PBFI  WHAT IS THE BEST APPROXIMATION THAT CAN BE MADE  WITH A GIVEN SET OF VECTORSENDITEMIZEIN THIS CHAPTER WE EXAMINE THE FIRST TWO QUESTIONS LEAVING THEREST OF THE QUESTIONS TO THE APPLICATIONS OF THE NEXT CHAPTERSUBSECTIONLINEAR INDEPENDENCEWE WILL FIRST EXAMINE THE QUESTION OF THE UNIQUENESS OF THEREPRESENTATION AS A LINEAR COMBINATIONBEGINDEFINITION  LABELDEFLININD  LET S BE A VECTOR SPACE AND LET T BE A SUBSET OF S  THE SET  T IS BF LINEARLY INDEPENDENT IF FOR EACH FINITE NONEMPTY SUBSET  OF T SAY PBF1PBF2LDOTSPBFM THE ONLY SET OF  SCALARS SATISFYING THE EQUATION C1PBF1  C2PBF2  LDOTS  CMPBFM  0 IS THE TRIVIAL SOLUTION C1  C2  CDOTS  CM 0 INDEXLINEARLY INDEPENDENT  THE SET OF VECTORS PBF1PBF2LDOTSPBFM IS SAID TO BE BF   LINEARLY DEPENDENT IF THERE EXISTS A SET OF SCALAR COEFFICIENTS C1C2LDOTSCM WHICH ARE NOT ALL ZERO SUCH THAT C1PBF1  C2PBF2  LDOTS  CMPBFM  0 ENDDEFINITIONBEGINEXAMPLE   BEGINENUMERATE  ITEM THE FUNCTIONS P1T P2T P3T P4T IN S OF    EXAMPLE REFEXMLINCOMB1 ARE LINEARLY DEPENDENT BECAUSE P4T  P1T  P3T  0THAT IS THERE IS A NONZERO LINEAR COMBINATION OF THE FUNCTIONS WHICHIS EQUAL TO ZEROITEM THE VECTORS PBF1  234T PBF2  162 AND  PBF3  162T ARE LINEARLY DEPENDENT SINCE 4PBF1  5PBF2  3PBF3  0ITEM THE FUNCTIONS P1T  T AND P2T  1T ARE LINEARLY  INDEPENDENT  ENDENUMERATEENDEXAMPLEBEGINDEFINITION  LET T BE A SET OF VECTORS IN A VECTOR SPACE S OVER A SET OF  SCALARS R THE NUMBER OF VECTORS IN T COULD BE INFINITE  THE  SET OF VECTORS V THAT CAN BE REACHED BY ALL POSSIBLE FINITE  LINEAR COMBINATIONS OF VECTORS IN T IS THE BF SPAN OF THE  VECTORS  THIS IS DENOTED BY V  LSPANTTHAT IS FOR ANY XBF IN V THERE IS SOME SET OF COEFFICIENTSCI IN R SUCH THAT XBF  SUMI1M CI PBFI WHERE EACH PBFI IN TENDDEFINITIONIT MAY BE OBSERVED THAT V IS A SUBSPACE OF S  WE ALSO OBSERVETHAT V  LSPANT IS THE SMALLEST SUBSPACE OF S CONTAINING TIN THE SENSE THAT FOR EVERY SUBSPACE MSUBSET S SUCH THAT T SUBSETM THEN V SUBSET MTHE SPAN OF A SET OF VECTORS CAN BE THOUGHT OF AS A LINE IF ITOCCUPIES ONE DIMENSION OR AS A PLANE IF IT OCCUPIES TWO DIMENSIONSOR AS A HYPERPLANE IF IT OCCUPIES MORE THAN TWO DIMENSIONS  IN THISBOOK WE WILL SPEAK OF THE EM PLANE SPANNED BY A SET REGARDLESS OFITS DIMENSIONALITYBEGINEXAMPLE BEGINENUMERATEITEM LET PBF1  110T AND PBF2  010T BE IN  RBB3  LINEAR COMBINATIONS OF THESE VECTORS ARE XBF  BEGINBMATRIXC1C2  C2  0 ENDBMATRIXFOR C1C2 IN RBB  THE SPACE U  LSPANPBF1PBF2 IS ASUBSET OF THE SPACE RBB3 IT IS THE PLANE IN WHICH THE VECTORS110T AND 010T LIE WHICH IS THE XY PLANE IN THE USUALCOORDINATE SYSTEM AS SHOWN IN FIGURE REFFIGPLAN1BEGINFIGUREHTBP  BEGINCENTERINPUTPICTUREDIRPLAN1  CAPTIONA SUBSPACE OF RBB3  LABELFIGPLAN1ENDCENTERENDFIGUREITEM LET P1T  1  T AND P2T  T  THEN V   LSPANP1P2 IS THE SET OF ALL POLYNOMIALS UP TO DEGREE 1  THE  SET V COULD BE ENVISIONED ABSTRACTLY AS A PLANE LYING IN THE  SPACE OF ALL POLYNOMIALSENDENUMERATEENDEXAMPLEBEGINDEFINITION  LET  T BE A SET OF VECTORS IN A VECTOR SPACE S AND LET V  SUBSET S BE A SUBSPACE  IF EVERY VECTOR XBF IN V  CAN BE WRITTEN AS A LINEAR COMBINATION OF VECTORS IN T THEN T  IS A BF SPANNING SET OF VENDDEFINITIONBEGINEXAMPLE   BEGINENUMERATE  ITEM THE VECTORS PBF1  1 6 5T PBF2  242T    PBF3  110T PBF4  752T FORM A SPANNING SET OF    RBB3  ITEM THE FUNCTIONS P1T  1T P2T  1T2 P3T     T2 AND P4T  2 FORM A SPANNING SET OF THE SET OF    POLYNOMIALS UP TO DEGREE 2  ENDENUMERATEENDEXAMPLELINEAR INDEPENDENCE PROVIDES US WITH WHAT WE NEED FOR A UNIQUEREPRESENTATION AS A LINEAR COMBINATION AS THE FOLLOWING THEOREMSHOWSBEGINTHEOREM LABELTHMUNIQBAS  LET S BE A VECTOR SPACE AND LET T BE A NONEMPTY SUBSET OF S  THE SET T IS LINEARLY INDEPENDENT IF AND ONLY IF FOR EACH NONZERO  XBF IN LSPANT THERE IS EXACTLY ONE FINITE SUBSET OF T  WHICH WE WILL DENOTE AS PBF1PBF2LDOTSPBFM AND A  UNIQUE SET OF SCALARS C1C2LDOTSCM SUCH THAT XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFMENDTHEOREMBEGINPROOF  WE WILL FIRST SHOW THAT T LINEARLY INDEPENDENT IMPLIES A UNIQUE  REPRESENTATION  SUPPOSE THAT THERE ARE TWO SETS OF VECTORS IN T PBF1 PBF2 LDOTS PBFM QQUAD TEXTANDQQUAD    QBF1 QBF2 LDOTS QBFNAND CORRESPONDING NONZERO COEFFICIENTS SUCH THAT XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFM QQUADTEXTANDQQUADXBF  D1 QBF1  D2 QBF2  CDOTS  DN QBFNWE NEED TO SHOW THAT NM AND PBFI  QBFI FOR I12LDOTSMAND THAT CI  DI  WE NOTE THAT C1 PBF1  C2 PBF2  CDOTS  CM PBFM  D1 QBF1  D2QBF2  CDOTS  DN QBFN  0SINCE C1 NEQ 0 BY THE DEFINITION OF LINEAR INDEPENDENCE THEVECTOR PBF1 MUST BE AN ELEMENT OF THE SET QBF1QBF2LDOTSQBFN AND THE CORRESPONDING COEFFICIENTS MUST BE EQUAL SAYPBF1  QBF1 AND C1  D1  SIMILARLY SINCE C2 NEQ 0 WECAN SAY THAT PBF2  QBF2 AND C2  D2  PROCEEDING SIMILARLYWE MUST HAVE PBFIQBFI FOR I12LDOTSM AND CI  DI  CONVERSELY SUPPOSE THAT FOR EACH XBF IN LSPANT THEREPRESENTATION XBF  C1 PBF1  CDOTS CM PBFM IS UNIQUEASSUME TO THE CONTRARY THAT T IS LINEARLY DEPENDENT SO THAT THEREARE VECTORS PBF1PBF2LDOTS PBFM SUCH THATBEGINEQUATION PBF1   A2PBF2 A3 PBF3  CDOTS  AM  PBFMLABELEQLININD2ENDEQUATIONBUT THIS GIVES TWO REPRESENTATIONS OF THE VECTOR PBF1 ITSELF ANDTHE LINEAR COMBINATION  REFEQLININD2  SINCE THIS CONTRADICTSTHE UNIQUE REPRESENTATION T MUST BE LINEARLY INDEPENDENTENDPROOFSUBSECTIONBASIS AND DIMENSIONLABELSECHAMELBASISUP TO THIS POINT WE HAVE USED THE TERM DIMENSION FREELY ANDWITHOUT A FORMAL DEFINITION  WE HAVE NOT CLARIFIED WHAT IS MEANT BYFINITEDIMENSIONAL AND INFINITEDIMENSIONAL VECTOR SPACESIN THIS SECTION WE AMEND THIS OMISSION BY DEFINING THE HAMEL BASISOF A VECTOR SPACE  BEGINDEFINITION  INDEXHAMEL BASIS  LET S BE A VECTOR SPACE AND LET T BE A SET OF VECTORS FROM S  SUCH THAT LSPANT  S  IF T IS LINEARLY INDEPENDENT THEN T  IS SAID TO BE A BF HAMEL BASIS FOR SENDDEFINITIONBEGINEXAMPLE BEGINENUMERATEITEM THE SET OF VECTORS IN THE LAST EXAMPLE IS NOT LINEARLY  INDEPENDENT SINCE   4 PBF1  5 PBF2 21 PBF3  5 PBF4  0HOWEVER THE SET T  PBF1PBF2PBF3 IS LINEARLYINDEPENDENT AND SPANS THE SPACE RBB3  HENCE T IS A HAMELBASIS FOR RBB3ITEM THE VECTORS EBF1  BEGINBMATRIX1  0  0 ENDBMATRIXQQUADEBF2  BEGINBMATRIX 0  1  0  ENDBMATRIXQQUADEBF3  BEGINBMATRIX 0  0  1 ENDBMATRIXFORM ANOTHER HAMEL BASIS FOR RBB3  THIS BASIS IS OFTEN CALLEDTHE BF NATURAL BASISITEM THE VECTORS P1T  1 P2TT  P3T  T2 FORM A  HAMEL BASIS FOR THE SET S  MBOXALL POLYNOMIALS OF DEGREE LEQ 2ANOTHER HAMEL BASIS FOR S IS THE SET OF POLYNOMIALS Q1T  2Q2T  TT2 Q3T  TENDENUMERATEENDEXAMPLEAS THIS EXAMPLE SHOWS THERE IS NOT NECESSARILY A UNIQUE HAMEL BASISFOR A VECTOR SPACE  HOWEVER THE FOLLOWING THEOREM SHOWS THAT EVERYBASIS FOR A VECTOR SPACE HAVE A COMMON ATTRIBUTE THE CARDINALITY ORNUMBER OF ELEMENTS IN THE BASISBEGINTHEOREM LABELTHMBASISSAME  IF T1 AND T2 ARE HAMEL BASES FOR A VECTOR SPACE S THEN  T1 AND T2 HAVE THE SAME CARDINALITYENDTHEOREMTHE PROOF OF THIS THEOREM IS SPLIT INTO TWO PIECES THEFINITEDIMENSIONAL CASE AND THE INFINITEDIMENSIONAL CASE  THELATTER MAY BE OMITTED ON A FIRST READINGBEGINPROOF  FINITEDIMENSIONAL CASE SUPPOSE  T1  PBF1PBF2LDOTS PBFMQQUADTEXTANDQQUAD T2  QBF1QBF2LDOTSQBFNARE TWO HAMEL BASES OF S  EXPRESS THE POINT QBF1 IN T2 AS QBF1  C1 PBF1  C2 PBF2  CDOTS  CM PBFMAT LEAST ONE OF THE COEFFICIENTS CI MUST BE NONZERO LET US TAKETHIS AS C1  WE CAN THEN WRITE PBF1  FRAC1C1QBF1  C2 PBF2  CDOTS  CM PBFMBY THIS MEANS WE CAN ELIMINATE PBF1 AS A BASIS VECTOR IN T1 ANDUSE INSTEAD THE SET QBF1ALLOWBREAK PBF2ALLOWBREAK LDOTSALLOWBREAK PBFM AS A BASISSIMILARLY WE WRITE QBF2  D1 QBF1  D2 PBF2  CDOTS  DM PBFMAND AS BEFORE ELIMINATE PBF2 SO THAT  QBF1QBF2PBF3LDOTS PBFM FORMS A BASIS  CONTINUING IN THIS WAY WECAN ELIMINATE EACH PBFI SHOWING THAT QBF1 LDOTS QBFMSPANS THE SAME SPACE AS PBF1 LDOTS PBFM  WE CAN CONCLUDETHAT M GEQ N  SUPPOSE TO THE CONTRARY THAT N  M  THEN AVECTOR SUCH AS QBFM1 WHICH DOES NOT FALL IN THE BASIS SETQBF1LDOTS QBFM WOULD HAVE TO BE LINEARLY DEPENDENT WITHTHAT SET WHICH VIOLATES THE FACT THAT T2 IS ITSELF A BASISREVERSING THE ARGUMENT WE FIND THAT N GEQ M  IN COMBINATIONTHEN WE CONCLUDE THAT MNINFINITEDIMENSIONAL CASE  LET T1 AND T2 BE BASES  FOR ANXBF IN T1 LET T2XBF DENOTE THE UNIQUE FINITE SET OF POINTSIN T2 NEEDED TO EXPRESS XBF  CLAIM IF YBF IN T2 THEN YBF IN T2XBF FOR SOME XBF INT1  PROOF SINCE A POINT YBF IS IN S THEN YBF MUST BE AFINITE LINEAR COMBINATION OF VECTORS IN T1 SAY YBF  C1 XBF1  C2 XBF2  CDOTS  CM XBFMFOR SOME SET OF VECTORS XBFI IN T1  THEN FOR EXAMPLE XBF1  FRAC1C1YBF  C2 XBF2  CDOTS  CM XBFMSO THAT BY THE UNIQUENESS OF THE REPRESENTATION YBF IN B2XBFSINCE FOR EVERY YBF IN T2 THERE IS SOME XBF IN T1 SUCH THAT YBFIN T2XBF IT FOLLOWS THAT T2  BIGCUPXBF IN T1 T2XBFNOTING THAT THERE ARE T1  INDEX  CDOT INDEXBAR CDOT  SETS IN THIS UNIONFOOTNOTERECALL THAT THE NOTATION S  INDICATES THE CARDINALITY OF THE SET S SEE SECTION  REFSECFUNDAMENTALS EACH OF WHICH CONTRIBUTES AT LEAST ONE  ELEMENT TO T2 WE CONCLUDE THAT T2 GEQ T1  NOW TURNING THE ARGUMENT AROUND WE CONCLUDE THAT T1 GEQ  T2  BY THESE TWO INEQUALITIES WE CONCLUDE THAT T1  T2ENDPROOFON THE STRENGTH OF THIS THEOREM WE CAN STATE A CONSISTENT DEFINITIONFOR THE DIMENSION OF A VECTOR SPACEBEGINDEFINITION  LET T BE A HAMEL BASIS FOR A VECTOR SPACE S  THE CARDINALITY OF  T IS THE BF DIMENSION OF S  THIS IS DENOTED AS  DIMENSIONS  IT IS THE EM NUMBER OF LINEARLY INDEPENDENT    VECTORS REQUIRED TO SPAN THE SPACEENDDEFINITIONSINCE THE DIMENSION OF A VECTOR SPACE IS UNIQUE WE CAN CONCLUDE THATA BASIS T FOR A SUBSPACE S IS A EM SMALLEST SET OF VECTORSWHOSE LINEAR COMBINATIONS CAN FORM EVERY VECTOR IN A VECTOR SPACE SIN THE SENSE THAT A BASIS OF T VECTORS IS CONTAINED IN EVERY OTHERSPANNING SET FOR STHE LAST REMAINING FACT WHICH WE WILL NOT PROVE SHOWS THE IMPORTANCEOF THE HAMEL BASIS EM EVERY VECTOR SPACE HAS A HAMEL BASIS  SOFOR MANY PURPOSES WHATEVER WE WANT TO DO WITH A VECTOR SPACE CAN BEDONE TO THE HAMEL BASISBEGINEXAMPLE  LET S BE THE SET OF ALL POLYNOMIALS  THEN A POLYNOMIAL XT IN  S CAN BE WRITTEN AS A LINEAR COMBINATION OF THE FUNCTIONS  1TT2LDOTS  IT CAN BE SHOWN SEE EXERCISE  REFEXLINIDPOLY THAT THIS SET OF FUNCTIONS IS LINEARLY  INDEPENDENT  HENCE THE DIMENSION OF S IS INFINITEENDEXAMPLEBEGINEXAMPLE  CITEFRIEDMAN TO ILLUSTRATE THAT INFINITE DIMENSIONAL VECTOR  SPACES CAN BE DIFFICULT TO WORK WITH AND THAT PARTICULAR CARE IS  REQUIRED WE DEMONSTRATE THAT FOR AN INFINITEDIMENSIONAL VECTOR  SPACE S AN INFINITE SET OF LINEARLY INDEPENDENT VECTORS WHICH  SPAN S NEED NOT FORM A BASIS FOR S    LET X BE THE INFINITESEQUENCE SPACE WITH ELEMENTS OF THE FORM  X1X2X3LDOTS WHERE EACH XI IN RBB  THE SET OF  VECTORS PBFJ  100LDOTS010LDOTS QQUAD J23LDOTSWHERE THE SECOND 1 IS IN THE JTH POSITION FORMS A SET OF LINEARLYINDEPENDENT VECTORS  WE FIRST SHOW THE SET PBFJJ23LDOTS SPANS X  LET X X1X2X3LDOTS BE AN ARBITRARY ELEMENT OF X  LET  SIGMAN  X1  X2  CDOTS  XNAND LET TAUN BE AN INTEGER LARGER THAN NSIGMAN2  NOWCONSIDER THE SEQUENCE OF VECTORS YBFN  X2 PBF2  X3 PBF3  CDOTS  XN PBFN FRACSIGMANTAUNPBFN1  CDOTS  PBFPWHERE P  NTAUN  FOR EXAMPLE BEGINALIGNEDYBF3  XBF2 PBF2  XBF3 PBF3  FRACX1  X2X3TAUNPBF4  PBF5  CDOTS  PBF4TAUN  XBF2 PBF2  XBF3 PBF3  X1 X2X31FRAC1TAUNFRAC1TAUNLDOTSFRAC1TAUNENDALIGNED IN THE LIMIT AS N RIGHTARROW INFTY THE RESIDUAL TERM BECOMES X1  X2  CDOTS100LDOTSAND YBFNRIGHTARROW XBF  SO THERE IS A REPRESENTATION FOR XBFUSING THIS INFINITE SET OF BASIS FUNCTIONSHOWEVER  THIS IS THE SUBTLE BUT IMPORTANT POINT  THEREPRESENTATION EXISTS AS A RESULT OF A LIMITING PROCESS  THERE IS NOEM FINITE SET OF FIXED SCALARS C2 C3 LDOTS CN SUCH THATTHE SEQUENCE XBF  100LDOTS CAN BE WRITTEN IN TERMS OF THEBASIS FUNCTIONS AS XBF  100LDOTS  C2 PBF2  C3 PBF3  LDOTS  CN PBFNWHEN WE INTRODUCED THE CONCEPT OF LINEAR COMBINATIONS IN DEFINITIONREFDEFLC ONLY EM FINITE SUMS WERE ALLOWED  SINCE REPRESENTINGXBF WOULD REQUIRE AN INFINITE SUM THE SET OF FUNCTIONSPBF2PBF3 LDOTS DOES EM NOT FORM A BASISIT MAY BE OBJECTED THAT IT WOULD BE STRAIGHTFORWARD TO SIMPLY EXPRESSAN INFINITE SUM SUMJ2INFTY C2 PBF2 AND HAVE DONE WITH THEMATTER  BUT DEALING WITH INFINITE SERIES ALWAYS REQUIRES MORE CARETHAN DOES FINITE SERIES SO WE CONSIDER THIS AS A DIFFERENT CASEENDEXAMPLESUBSECTIONFINITEDIMENSIONAL VECTOR SPACES AND MATRIX NOTATIONTHE MAJOR FOCUS OF OUR INTEREST IN VECTOR SPACES WILL BE ONFINITEDIMENSIONAL VECTOR SPACES  EVEN WHEN DEALING WITHINFINITEDIMENSIONAL VECTOR SPACES WE SHALL FREQUENTLY BE INTERESTEDIN FINITEDIMENSIONAL REPRESENTATIONS  IN THE CASE OFFINITEDIMENSIONAL VECTOR SPACES EM WE SHALL REFER TO THE HAMEL  BASIS SIMPLY AS THE BASISONE PARTICULARLY USEFUL ASPECT OF FINITEDIMENSIONAL VECTOR SPACES ISTHAT MATRIX NOTATION CAN BE USED FOR CONVENIENT REPRESENTATION OFLINEAR COMBINATIONS  LET THE MATRIX A BE FORMED BY STACKING THEVECTORS PBF1 PBF2 LDOTSPBFM SIDE BY SIDE  A  BEGINBMATRIXPBF1 PBF2 CDOTSPBFM ENDBMATRIXFOR A VECTOR CBF  BEGINBMATRIXC1  C2  VDOTS  CM ENDBMATRIXTHE PRODUCT XBF  ACBF COMPUTES THE LINEAR COMBINATION XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFMTHE QUESTION OF THE LINEAR DEPENDENCE OF THE VECTORS PBFI CANBE EXAMINED BY LOOKING AT THE RANK OF THE MATRIX A AS DISCUSSED INSECTION REFSECRANKBEGINEXERCISES  ITEM AN EQUIVALENT DEFINITION FOR LINEAR INDEPENDENCE FOLLOWS  A    SET T IS LINEARLY INDEPENDENT IF FOR EACH VECTOR XBF IN T    XBF IS NOT A LINEAR COMBINATION OF THE POINTS IN THE SET T      XBF THAT IS THE SET T WITH THE VECTOR XBF REMOVED          SHOW THAT THIS DEFINITION IS EQUIVALENT TO THAT OF DEFINITION    REFDEFLININD  ITEM LET S BE A FINITEDIMENSIONAL VECTOR SPACE WITH    DIMENSIONS  M  SHOW THAT EVERY SET CONTAINING M1 POINTS    IS LINEARLY DEPENDENT  ITEM SHOW THAT IF T IS A SUBSET OF A VECTOR SPACE S WITH    LSPANTS SHOW THAT T CONTAINS A HAMEL BASIS OF S  ITEM LET S DENOTE THE SET OF ALL SOLUTIONS OF THE DIFFERENTIAL    EQUATION DEFINED ON C30INFTY  SEE DEFINITION    REFDEFCLASSCK  INDEXCKCLASS CK FRACD3 XDT3  B FRACDX2DT2  C FRACDXDT  DX     0SHOW THAT S HAS DIMENSION 3  ITEM LET S BE L202PI AND LET T BE THE SET OF ALL    FUNCTIONS XNT  EJNT FOR N01LDOTS  SHOW THAT T IS    LINEARLY INDEPENDENT      CONCLUDE THAT L202PI IS AN    INFINITE DIMENSIONAL SPACE    HINT ASSUME THAT C1 EJ N1 T  C2    EJ N2 T  CDOTS  CM EJ NM T  0  DIFFERENTIATE    M1 TIMES  ITEM LABELEXLINIDPOLY SHOW THAT THE SET 1TT2LDOTSTM    IS A LINEARLY INDEPENDENT SET HINT THE FUNDAMENTAL THEOREM OF    ALGEBRA STATES THAT A POLYNOMIAL FX OF DEGREE M HAS EXACTLY    M ROOTS COUNTING MULTIPLICITYKEENER P 3ENDEXERCISESSECTIONNORMS AND NORMED VECTOR SPACESLABELSECNORMVSWHEN DEALING WITH VECTOR SPACES IT IS COMMON TO TALK ABOUT THE LENGTHAND DIRECTION OF THE VECTOR AND THERE IS AN INTUITIVE GEOMETRICCONCEPT AS TO WHAT THE LENGTH AND DIRECTION ARE  THERE ARE A VARIETYOF WAYS OF DEFINING THE LENGTH OF A VECTOR  THE MATHEMATICAL CONCEPTASSOCIATED WITH THE LENGTH OF A VECTOR IS THE BF NORM WHICH ISDISCUSSED IN THIS SECTION  IN SECTION REFSECINNERPROD1 WEINTRODUCE THE CONCEPT OF AN INNER PRODUCT WHICH IS USED TO PROVIDE ANINTERPRETATION OF ANGLE BETWEEN VECTORS AND HENCE DIRECTIONBEGINDEFINITION  LET S BE A VECTOR SPACE WITH ELEMENTS XBF  A REALVALUED  FUNCTION  XBF IS SAID TO BE A BF NORM INDEXNORM IF  XBF SATISFIES THE FOLLOWING PROPERTIES  BEGINENUMERATEN1  ITEM XBF GEQ 0 FOR ANY XBF IN S  ITEM XBF  0 IF AND ONLY IF XBF  ZEROBF  ITEM ALPHA XBF  ALPHA  XBF WHERE ALPHA IS AN    ARBITRARY SCALAR  ITEM XBF  YBF LEQ  XBF   YBF TRIANGLE    INEQUALITY  ENDENUMERATETHE REAL NUMBER  XBF IS SAID TO BE THE NORM OF XBF OR THELENGTH OF XBFENDDEFINITIONTHE TRIANGLE INEQUALITY N4 CAN BE INTERPRETED GEOMETRICALLY USINGFIGURE REFFIGTRIINEQ2 WHERE XBF YBF AND ZBF ARETHE SIDES OF A TRIANGLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRTRIINEQ2    CAPTIONA TRIANGLE INEQUALITY INTERPRETATION    LABELFIGTRIINEQ2  ENDCENTERENDFIGUREA NORM FEELS A LOT LIKE A METRIC BUT ACTUALLY REQUIRES MORESTRUCTURE THAN A METRIC  FOR EXAMPLE THE DEFINITION OF A NORMREQUIRES THAT ADDITION XBF  YBF AND SCALAR MULTIPLICATION ALPHAXBF ARE DEFINED WHICH WAS NOT THE CASE FOR A METRICNEVERTHELESS BECAUSE OF THEIR SIMILAR PROPERTIES NORMS AND METRICSCAN BE DEFINED IN TERMS OF EACH OTHER  FOR EXAMPLE IF  XBF ISA NORM THEN  DXBFYBF   XBF  YBFIS A METRIC  THE TRIANGLE INEQUALITY FOR METRICS IS ESTABLISHED BYNOTING THAT  XBF  YBF   XBF  ZBF  ZBF  XBF LEQ  XBF ZBF   YBF  ZBFTHIS TRICK OF ADDING AND SUBTRACTING THE QUANTITY TO MAKE THE ANSWERCOME OUT RIGHT IS OFTEN USED IN ANALYSIS  ALTERNATIVELY GIVEN AMETRIC D DEFINED ON A VECTOR SPACE A NORM CAN BE WRITTEN AS  XBF  DXBFZEROBFTHE DISTANCE THAT XBF IS FROM THE ORIGIN OF THE VECTOR SPACEBEGINEXAMPLE  BASED UPON THE METRICS WE HAVE ALREADY SEEN WE CAN READILY DEFINE  SOME USEFUL NORMS FOR NDIMENSIONAL VECTORS  BEGINENUMERATE  ITEM THE L1 NORM  XBF1  SUMI1N XI  ITEM THE L2 NORM XBFP  LEFTSUMI1N      XIPRIGHT1P  ITEM THE LINFTY NORM  XBFINFTY  MAXI12LDOTSN    XI  ENDENUMERATEEACH OF THESE NORMS INTRODUCES ITS OWN GEOMETRY  CONSIDER FOREXAMPLE THE UNIT SPHERE DEFINED BY SP   XBF IN RBB2MC  XBFP LEQ 1FIGURE REFFIGSPHERES ILLUSTRATES THE SHAPE OF SUCH SPHERES FORVARIOUS VALUES OF PENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRSPHERES    CAPTIONUNIT SPHERES IN RBB2 UNDER VARIOUS    PROTECTLPROTECTPPROTECT NORMS    LABELFIGSPHERES  ENDCENTERENDFIGUREBEGINEXAMPLE  WE CAN ALSO DEFINE NORMS OF FUNCTIONS DEFINED OVER THE INTERVAL  AB  BEGINENUMERATE  ITEM THE L1 NORM  XT1  INTAB XTDT  ITEM THE LP NORM XT2  LEFTINTAB XTPDT    RIGHT1P FOR 1 LEQ P  INFTY  ITEM THE LINFTY NORM XTINFTY  SUPT IN AB XT  ENDENUMERATEENDEXAMPLETHE LINFTY AND LINFTY NORMS ARE REFERRED TO AS THE EM  UNIFORM NORMSBEGINDEFINITION  A BF NORMED LINEAR SPACE IS A PAIR S CDOT WHERE S IS  A VECTOR SPACE AND  CDOT IS A NORM DEFINED ON S  A NORMED  LINEAR SPACE IS OFTEN DENOTED SIMPLY BY SENDDEFINITIONWHEN DISCUSSING THE METRICAL PROPERTIES OF A NORMED LINEAR SPACE THEMETRIC IS DEFINED IN TERMS OF THE NORM DXBFYBF  XBFYBFBEGINDEFINITION  A VECTOR XBF IS SAID TO BE BF NORMALIZED IF XBF    1 INDEXNORMALIZED VECTOR  IT IS POSSIBLE TO NORMALIZE ANY VECTOR EXCEPT THE ZERO VECTOR YBF   XBFXBF HAS YBF  1  A NORMALIZED VECTOR IS ALSO  REFERRED TO AS A BF UNIT VECTOR INDEXUNIT VECTORENDDEFINITIONWITH A VARIETY OF NORMS TO CHOOSE FROM IT IS NATURAL TO ADDRESS THEISSUE OF WHICH NORM SHOULD BE USED IN A PARTICULAR  OFTEN THE L2OR L2 NORM IS USED FOR REASONS WHICH BECOME CLEAR SUBSEQUENTLYHOWEVER OCCASIONS MAY ARISE IN WHICH OTHER NORMS OR NORMLIKEFUNCTIONS ARE USED  FOR EXAMPLE IN A HIGHSPEED SIGNALPROCESSINGALGORITHM IT MAY BE NECESSARY TO USE THE L1 NORM SINCE IT MAY BEEASIER IN THE AVAILABLE HARDWARE TO COMPUTE AN ABSOLUTE VALUE THAN ITIS TO COMPUTE A SQUARE  OR IN A PROBLEM OF DATA REPRESENTATION OFAUDIO INFORMATION QUANTIZATION IT MAY BE APPROPRIATE TO USE A NORMFOR WHICH THAT REPRESENTATION IS CHOSEN THAT IS BEST AS PERCEIVED BYHUMAN LISTENERS  IDEALLY A NORM THAT MEASURED EXACTLY THE DISTORTIONPERCEIVED BY THE HUMAN EAR WOULD BE DESIRED IN SUCH AN APPLICATIONTHIS IS ONLY APPROXIMATELY ACHIEVABLE SINCE IT DEPENDS UPON SO MANYPSYCHOACOUSTIC EFFECTS OF WHICH ONLY A FEW ARE UNDERSTOOD  SIMILARCOMMENTS COULD BE MADE REGARDING NORMS FOR VIDEO CODING  IN SHORTTHE NORM SHOULD BE CHOSEN THAT IS BEST SUITED TO THE PARTICULARAPPLICATIONTHE EXACT NORM VALUES COMPUTED FOR A VECTOR XBF CHANGE DEPENDING ONTHE PARTICULAR NORM USED BUT A VECTOR THAT IS SMALL WITH RESPECT TOONE NORM IS ALSO SMALL WITH RESPECT TO ANOTHER NORM  NORMS ARE THUSEQUIVALENT IN THE SENSE DESCRIBED IN THE FOLLOWING THEOREMBEGINTHEOREMNORM EQUIVALENCE THEOREM  IF  CDOT  AND   CDOT  ARE TWO NORMS ON RBBN OR CBBN THEN  XBFK  RIGHTARROW 0 TEXT AS  KRIGHTARROW INFTY QUADTEXTIF AND ONLY IF  QUAD  XBFK RIGHTARROW 0 TEXT AS  KRIGHTARROW INFTYENDTHEOREMTHE  PROOF OF THIS THEOREM MAKES USE OF THE CAUCHYSCHWARZ INEQUALITYWHICH IS INTRODUCED IN SECTION REFSECCS  YOU MAY WANT TO COMEBACK TO THIS PROOF AFTER READING THAT SECTIONBEGINPROOFIT SUFFICES TO SHOW THAT THERE ARE CONSTANTS C1 C2  0 SUCH THATBEGINEQUATIONC1  XBF  LEQ  XBF  LEQ C2  XBF LABELEQNORM2ENDEQUATIONTO PROVE REFEQNORM2 IT SUFFICES TO ASSUME THAT  CDOT IS THE L2 NORM  TO SEE THIS OBSERVE THAT IF D1  XBF  LEQ XBF2 LEQ D2  XBF QQUAD TEXTANDQQUAD D1  XBF  LEQ XBF2 LEQ D2  XBF THEN BEGINALIGND1XBF LEQ D2  XBF  INTERTEXTANDD1 XBF   LEQ D2  XBF ENDALIGNSO REFEQNORM2 HOLDS WITH C1  D1D2 AND C2  D2D1LET XBF BE EXPRESSED AS A LINEAR COMBINATION OF BASIS VECTORS XBF  SUMI1N XI EBFITHEN BY THE PROPERTIES OF THE NORM  XBF  LEFT SUMI1N XI EBFI RIGHT LEQ SUMI1NXI  EBFITHE SUM ON THE RIGHT IS SIMPLY THE INNER PRODUCT OF THE VECTORCOMPOSED OF THE MAGNITUDES OF THE XIS AND THE VECTOR COMPOSED OFTHE MAGNITUDES OF THE BASIS VECTORS  BEING AN INNER PRODUCT THECAUCHYSCHWARZ INEQUALITY APPLIES AND   XBF  LEQ  XBF 2LEFTSUMI1N  EBFI  2RIGHT12 LET BETA  LEFTSUMI1N  EBFI  2RIGHT12THEN THE LEFT INEQUALITY OF REFEQNORM2 APPLIES WITH C1 1BETAFOR POINTS XBF ON THE UNIT SPHERE S   XBF  XBF 2 1 THE NORM  CDOT  MUST BE GREATER THAN 0 BY THE PROPERTIES OFNORMS AND HENCE  XBF  GEQ ALPHA FOR SOME ALPHA  0 FORXBF IN S  THEN  XBF   LEFT FRACXBF XBF 2  RIGHT  XBF 2 GEQALPHA  XBF 2SO THE RIGHTHAND INEQUALITY HOLDS WITH C2  1ALPHAENDPROOFFOR EXAMPLEBEGINEQUATIONBEGINSPLIT XBF 2 LEQ XBF1 LEQ SQRTNXBF 2 XBF INFTY LEQ XBF2 LEQ SQRTNXBF INFTY XBF INFTY LEQ XBF1 LEQ NXBF INFTYENDSPLITLABELEQNORMCOMPENDEQUATIONFINALLY WE END WITH AN IMPORTANT DEFINITIONBEGINDEFINITION  FOR A SEQUENCE XN IN A NORMED LINEAR SPACE SPACE S CDOT   IF THERE EXISTS  A NUMBER M INFTY SUCH THAT XN  M QQUAD FORALL NTHEN THE SEQUENCE IS SAID TO BE BF BOUNDED INDEXBOUNDED SEQUENCEENDDEFINITION BEGINDEFINITION   A SEQUENCE XN IS BF MONOTONIC IF  X1 LEQ X2 LEQ X3 LEQ CDOTS  OR  X1 GEQ X2 GEQ X3 GEQ CDOTS  ENDDEFINITION FOR SEQUENCES OVER THE REAL NUMBERS THE FOLLOWING FACT IS CLEAR EVERY BOUNDED MONOTONIC SEQUENCE IS CONVERGENT  SINCE THE SEQUENCE IS BOUNDED THE MONOTONIC SEQUENCE RUNS OUT OF ROOM AND HENCE MUST HAVE A LIMIT POINT WHICH BECAUSE THE SEQUENCE IS MONOTONIC MUST BE UNIQUESUBSECTIONFINITEDIMENSIONAL NORMED LINEAR SPACESTHE NOTION OF A CLOSED SET AND A COMPLETE SET WERE INTRODUCED INSECTION REFSECSEQUENCES   AS POINTED OUT HAVING COMPLETE SETS ISADVANTAGEOUS BECAUSE ALL CAUCHY SEQUENCES CONVERGE SO THATCONVERGENCE OF A SEQUENCE CAN BE ESTABLISHED SIMPLY BY DETERMININGWHETHER A SEQUENCE IS CAUCHYFINITEDIMENSIONAL NORMED LINEAR SPACES HAVE SEVERAL VERY USEFUL PROPERTIESBEGINENUMERATEITEM EVERY FINITEDIMENSIONAL SUBSPACE OF A VECTOR SPACE IS CLOSEDITEM EVERY FINITEDIMENSIONAL SUBSPACE OF A VECTOR SPACE IS COMPLETEITEM IF LMC X RIGHTARROW Y IS A LINEAR OPERATOR AND X IS A  FINITE DIMENSIONAL NORMED VECTOR SPACE THEN L IS CONTINUOUS  THIS IS TRUE EVEN IF Y IS NOT FINITE DIMENSIONAL  AS WE SHALL  SEE IN CHAPTER REFCHAPMATINV THIS MEANS THAT THE OPERATOR IS  ALSO BOUNDEDITEM AS OBSERVED ABOVE DIFFERENT NORMS ARE EQUIVALENT ON RBBN OR  CBBN  IN FACT IN ANY FINITEDIMENSIONAL SPACE ANY TWO NORMS  ARE EQUIVALENTENDENUMERATEA LOT OF THE ISSUES OVER WHICH A MATHEMATICIAN WOULD FRET ENTIRELYDISAPPEAR IN FINITEDIMENSIONAL SPACES  THIS IS PARTICULARLY USEFULSINCE MANY OF THE PROBLEMS OF INTEREST IN SIGNAL PROCESSING ARE FINITEDIMENSIONALWE WILL NOT PROVE THESE USEFUL FACTS HERE  INTERESTED READERS SHOULDCONSULT FOR EXAMPLE CITESECTION 510NAYLORSELLBEGINEXERCISESITEM SHOW THAT IN A NORMED LINEAR SPACE BOXED X  Y LEQ XYITEM USING THE TRIANGLE INEQUALITY SHOW THAT ZX LEQ ZY  YXITEM LET P BE IN THE RANGE 0  P  1 AND CONSIDER THE SPACE  LP01 OF ALL FUNCTIONS WITH   X  INT01 XTPDT  INFTYSHOW THAT X IS NOT A NORM ON LP01  HOWEVER SHOW THATDXY   XY IS A METRIC  HINT FOR A REAL NUMBER ALPHASUCH THAT 0 LEQ ALPHA LEQ 1 NOTE THAT ALPHA LEQ ALPHAP LEQ1ITEM SHOW THAT THE NORM FUNCTION CDOTMC  S RIGHTARROW RBB  IS CONTINUOUS  HINT USE THE TRIANGLE INEQUALITYITEM SHOW THAT A NORM IS A CONVEX FUNCTION  SEE SECTION  REFSECCONVFUNCITEM FOR EACH OF THE INEQUALITY RELATIONSHIPS BETWEEN NORMS IN  REFEQNORMCOMP DETERMINE A VECTOR XBF FOR WHICH EACH  INEQUALITY IS ACHIEVED WITH EQUALITYENDEXERCISESSECTIONINNER PRODUCTS AND INNER PRODUCT SPACESLABELSECINNERPROD1AN INNER PRODUCT IS AN OPERATION ON TWO VECTORS THAT RETURNS A SCALARVALUE  INNER PRODUCTS CAN BE USED TO PROVIDE THE GEOMETRICINTERPRETATION OF THE DIRECTION OF A VECTOR IN AN ARBITRARY VECTORSPACE  THEY CAN ALSO BE USED TO DEFINE A NORM KNOWN AS THE INDUCEDNORMWE WILL DEFINE THE INNER PRODUCT IN THE GENERAL CASE IN WHICH THEVECTOR SPACE S HAS ELEMENTS THAT ARE COMPLEXBEGINDEFINITION  LET S BE A VECTOR SPACE DEFINED OVER A SCALAR  FIELD R  AN BF INNER PRODUCT IS A FUNCTION  LACDOTCDOTRAMC STIMES S RIGHTARROW R WITH THE FOLLOWING  PROPERTIES INDEXINNER PRODUCT  BEGINENUMERATEIP1  ITEM LA XBFYBFRA  OVERLINELA YBFXBFRA WHERE THE    OVERBAR INDICATES COMPLEX CONJUGATION  FOR VECTORS DEFINED OVER A    FIELD OTHER THAN COMPLEX NUMBERS THIS SIMPLIFIES TO LA    XBFYBFRA  LA YBFXBFRA  ITEM LA ALPHAXBFYBF RA  ALPHALA XBFYBFRA  ITEM LA XBFYBFZBFRA  LA XBFZBFRA  LA    YBFZBFRA  ITEM LA XBFXBFRA  0 IF XBF NEQ 0 AND LA XBFXBF    RA  0 IF AND ONLY IF XBF  0  ENDENUMERATEENDDEFINITIONBEGINDEFINITION    A VECTOR SPACE EQUIPPED WITH AN INNER PRODUCT IS CALLED AN BF  INNERPRODUCT SPACE   ENDDEFINITION  INNERPRODUCT SPACES ARE SOMETIMES CALLED PREHILBERT SPACES  WE  ENCOUNTER IN SECTION REFSECHILBERT WHAT A HILBERT SPACE ISTHERE ARE A VARIETY OF WAYS THAT AN INNER PRODUCT CAN BEDEFINED  NOTATIONAL ADVANTAGE AND ALGORITHMIC EXPEDIENCY CAN BEOBTAINED BY SUITABLE SELECTION OF AN INNER PRODUCT  WE BEGIN WITHTHE MOST STRAIGHTFORWARD EXAMPLES OF INNER PRODUCTSBEGINEXAMPLE  FOR FINITEDIMENSIONAL VECTORS XBF YBF IN RBBN THE  CONVENTIONAL INNER PRODUCT BETWEEN THE VECTORS XBF  BEGINBMATRIXX1  X2  VDOTS  XNENDBMATRIXQQUAD TEXTANDQQUAD YBF  BEGINBMATRIX  Y1  Y2  VDOTS   YNENDBMATRIXISBEGINALIGNEDLA XBFYBF RA  X1Y1  X2 Y1  CDOTS  XN YN  SUMI1N XI YI  YBFT XBF  XBFT YBFENDALIGNEDTHIS INNER PRODUCT IS THE BF EUCLIDEAN INNER PRODUCT THIS IS ALSOTHE BF DOT PRODUCT INDEXDOT PRODUCTSEEINNER PRODUCT USED INVECTOR CALCULUS AND IS SOMETIMES WRITTEN LA XBFYBFRA  XBFCDOT YBFIF THE VECTORS ARE IN CBBN WITH COMPLEX ELEMENTS THEN THEEUCLIDEAN INNER PRODUCT IS LA XBFYBFRA  SUMK1N XK OVERLINEYK  YBFH XBF ENDEXAMPLEBEGINEXAMPLE  EXTENDING THE SUM OF PRODUCTS IDEA TO FUNCTIONS THE  FOLLOWING IS AN INNER PRODUCT FOR THE SPACE OF FUNCTIONS DEFINED ON  01  LA XTYTRA  INT01 XTOVERLINEYT DT FOR FUNCTIONS DEFINED OVER RBB AN INNER PRODUCT IS LA XTYTRA  INTINFTYINFTY XTYT DT ENDEXAMPLEBEGINEXAMPLE  CONSIDER A CAUSAL SIGNAL XT WHICH IS PASSED THROUGH A CAUSAL  FILTER WITH IMPULSE RESPONSE HT  THE OUTPUT AT A TIME  T IS YT  XTHTBIGGTT  INT0T XTAUHTTAUDTAULET GTAU  HTTAU  THEN YT  INT0T XTAU GTAU  DTAU  LA X G RAWHERE THE INNER PRODUCT IS LA FG RA  INT0T FTGT  DTSO THE OPERATION OF FILTERING AND TAKING THE OUTPUT AT A FIXED TIMEIS EQUIVALENT TO COMPUTING AN INNER PRODUCTENDEXAMPLEAN INNER PRODUCT CAN ALSO BE DEFINED ON MATRICES  LET S BE THEVECTOR SPACE OF MATSIZEMN MATRICES  THEN WE CAN DEFINE ANINNER PRODUCT ON THIS VECTOR SPACE BY LA AB RA  TRACEAH BBEGINEXERCISES KEENER P 8ITEM COMPUTE THE INNER PRODUCTS LA FG RA FOR THE FOLLOWINGUSING THE DEFINITION LA FG RA  INT01 FTGTDTBEGINENUMERATEITEM FT  T2  2T GT  T1ITEM FT  ET GT T1ITEM FT  COS2PI T GT  SIN2PI TENDENUMERATEITEM COMPUTE THE INNER PRODUCTS XBFT YBF OF THE FOLLOWING USING  THE EUCLIDEAN INNER PRODUCT  BEGINENUMERATE  ITEM XBF  1234T YBF  2341T  ITEM XBF  23 YBF  12T  ENDENUMERATEENDEXERCISESSUBSECTIONWEAK CONVERGENCEPROTECTFOOTNOTETHE CONCEPTS IN THIS SECTION ARE  USED BRIEFLY IN SECTION  REFSECORTHOSUB AND MOSTLY IN CHAPTER REFCHAPCOMPMAP IT ISRECOMMENDED THAT THIS SECTION BE SKIPPED ON A FIRST READING CHECK THE COMMENTEDOUT STUFF IN COMPMAPTEX IN ITERWHEN WE HAVE A SEQUENCE OF VECTORS XBFN AS WE SAW IN SECTIONREFSECSEQUENCES WE CAN TALK ABOUT CONVERGENCE OF THE SEQUENCETO SOME VALUE SAY XBFN RIGHTARROW XBF WHICH MEANS THAT  XBFN  XBF RIGHTARROW 0FOR SOME NORM  CDOT   IT IS INTERESTING TO EXAMINE THEQUESTION OF CONVERGENCE IN THE CONTEXT OF INNER PRODUCTSBEGINLEMMA LABELLEMCONTIP THE INNER PRODUCT IS CONTINUOUS  THAT IS IF XBFN RIGHTARROW XBF IN SOME INNER PRODUCT SPACE S THEN  LA XBFNYBFRA RIGHTARROW LA XBFYBFRA FOR ANY YBF IN SENDLEMMABEGINPROOF  SINCE XBFN IS CONVERGENT IT MUST BE BOUNDED SO THAT   XBFN LEQ M  INFTY  THEN BEGINALIGNED LA XBFNYBF RA  LA XBFYBFRA   LA XBFNXBF YBF RA  LEQ  XBFN  XBF   YBFENDALIGNEDSINCE  XBFN  XBF RIGHTARROW 0 THE CONVERGENCE OF LAXBFNYBFRA  IS ESTABLISHEDENDPROOFFROM THIS WE NOTE THAT CONVERGENCE XBFN RIGHTARROW XBF CALLEDEM STRONG CONVERGENCE IMPLIES LA XBFNYBF RA RIGHTARROWLA XBFYBFRA WHICH IS CALLED EM WEAK CONVERGENCE  ON THE OTHERHAND IT DOES NOT FOLLOW NECESSARILY THAT IF A SEQUENCE CONVERGESWEAKLY SO THAT INDEXSTRONG CONVERGENCE INDEXWEAK CONVERGENCE LA XBFN YBF RA RIGHTARROW LAXBFYBFRATHAT IT ALSO CONVERGES STRONGLYBEGINEXAMPLE  LET XBFN  000LDOTS100LDOTS BE THE SEQUENCE THAT IS  ALL 0 EXCEPT FOR A 1 AT POSITION N AND LET YBF   1121418LDOTS  THEN LA XBFN YBFRA RIGHTARROW 0BUT THE SEQUENCE  XBFN HAS NO LIMIT  THE SEQUENCE THUSCONVERGES WEAKLY BUT NOT STRONGLYENDEXAMPLEBEGINEXERCISESITEM LABELEXSTRCON SHOW THAT STRONG CONVERGENCE IMPLIES WEAK  CONVERGENCE  849 82 ENDEXERCISESSECTIONINDUCED NORMSLABELSECINDNORMWE HAVE SEEN THAT THE EUCLIDEAN NORM OF A VECTOR XBF IN RBBN IS DEFINED AS XBF22  X12  X22  CDOTS  XN2WE OBSERVE THAT THE INNER PRODUCT OF XBF WITH ITSELF IS LA XBFXBFRA  X12  X22  CDOTS  XN2HENCE WE CAN USE THE INNER PRODUCT TO PRODUCE A SPECIAL NORM CALLEDTHE BF INDUCED NORM  MORE GENERALLY GIVEN AN INNER PRODUCT LACDOTCDOTRA IN A VECTOR SPACE S WE HAVE THE INDUCED NORMDEFINED BY BOXED XBF   LA XBFXBFRA12 FOR EVERY X IN SIT SHOULD BE POINTED OUT THAT NOT EVERY NORM IS AN INDUCED NORM  FOREXAMPLE THE LP AND LP NORMS ARE ONLY INDUCED NORMS WHEN P2BEGINEXAMPLE  ANOTHER EXAMPLE OF AN INDUCED NORM IS FOR FUNCTIONS IN L2AB  XT2  LA XTXTRA12  LEFTINTAB XT2DTRIGHT12ENDEXAMPLEFOR AN INDUCED NORM WE HAVE THE FOLLOWING USEFUL FACT FOR AN INNERPRODUCT OVER A COMPLEX VECTOR SPACE BEGINALIGNED XBF  YBF2  LA XBFYBFXBFYBFRA  LA XBF XBF RA LA XBFYBFRA  LA YBFXBF RA  LA YBFYBFRA   XBF2  2 REAL LA XBFYBF RA  YBF2ENDALIGNEDFOR A VECTOR OVER A REAL VECTOR SPACE THIS SIMPLIFIES TO  XBF  YBF2   XBF2  2 LA XBFYBFRA  YBF2SECTIONTHE CAUCHYSCHWARZ INEQUALITYLABELSECCSIN THE DEFINITION OF A NORM ONE OF THE KEY REQUIREMENTS OF THEFUNCTION  CDOT IS THAT  XBF  YBF LEQ  XBF   YBFUP TO THIS POINT WE HAVE ASSUMED THAT THE METRICS MENTIONED DOSATISFY THIS PROPERTY  WE ARE NOW READY TO PROVE THIS RESULT FOR THEIMPORTANT SPECIAL CASE OF THE L2 OR L2 NORM OR MORE GENERALLYFOR A NORM INDUCED FROM ANY INNER PRODUCT  IN THE INTEREST OFGENERALITY WE SHALL EXPRESS THIS RESULT IN TERMS OF INNER PRODUCTSFIRSTTHE KEY INEQUALITY IN OUR PROOF IS THE EM CAUCHYSCHWARZ  INEQUALITY  INDEXCAUCHYSCHWARZ INEQUALITYINDEXINEQUALITIESCAUCHYSCHWARZ THIS INEQUALITY WILL PROVE TO BEONE OF THE CORNERSTONES OF SIGNAL PROCESSING ANALYSIS  IT WILLPROVIDE THE BASIS FOR THE IMPORTANT PROJECTION THEOREM AND BE THE KEYSTEP IN THE DERIVATION OF THE MATCHED FILTER  IT CAN BE USED TO PROVETHE IMPORTANT GEOMETRICAL FACT THAT THE GRADIENT OF A FUNCTION POINTSIN THE DIRECTION OF STEEPEST INCREASE WHICH IS THE KEY FACT USED INTHE DEVELOPMENT OF GRADIENT DESCENT OPTIMIZATION TECHNIQUES  NOT ONLYIS IT SPECIFICALLY USEFUL BUT THE ANALYSIS AND OPTIMIZATION PERFORMEDUSING THE CAUCHYSCHWARZ INEQUALITY PROVIDES A POWERFUL ARCHETYPE FORMANY OTHER OPTIMIZATION PROBLEMS OPTIMIZING VALUES CAN OFTEN BEOBTAINED BY ESTABLISHING AN INEQUALITY THEN SATISFYING THE CONDITIONSFOR WHICH THE INEQUALITY ACHIEVES EQUALITY  IF THE CAUCHYSCHWARZINEQUALITY DOES NOT SERVE THE PURPOSE OTHER INEQUALITIES OFTEN WILLSUCH AS THE CAUCHYSCHWARZS BIG BROTHERS THE HOLDER AND MINKOWSKIINEQUALITIESBEGINTHEOREM LABELTHMCS CAUCHYSCHWARZ INEQUALITY  IN AN  INNER PRODUCT SPACE S WITH INDUCED NORM CDOTBEGINEQUATIONBOXED LA XBFYBFRA LEQ  XBF  YBFLABELEQSW1ENDEQUATIONFOR ANY XBF YBF IN S WITH EQUALITY IF AND ONLY IF YBF ALPHA XBF FOR SOME ALPHAENDTHEOREMBEGINPROOF  BY EXPRESSING OUR PROOF IN TERMS OF INNER PRODUCTS WE  COVER BOTH THE CASE OF FINITE AND INFINITEDIMENSIONAL VECTORS  FOR GENERALITY WE ASSUME COMPLEX VECTORS    FIRST NOTE THAT IF XBF  0 OR YBF0 THE THEOREM IS TRIVIAL  SO WE EXCLUDE THESE CASES  FORM THE QUANTITYBEGINEQUATION  XBF  ALPHA YBF 2  XBF   2 REALLA  XBFALPHA YBFRA  ALPHA2  YBF 2LABELEQSW2ENDEQUATIONTHIS IS ALWAYS POSITIVE  WE WANT TO CHOOSE ALPHA TO MAKE THIS ASSMALL AS POSSIBLE  FOR REAL VECTORS THIS CAN BE DONE SIMPLY BYTAKING THE DERIVATIVE WITH RESPECT TO ALPHA AND EQUATING THEDERIVATIVE TO ZERO  WE DEMONSTRATE ANOTHER TECHNIQUE BY COMPLETINGTHE SQUARE INDEXCOMPLETING THE SQUARE SEE APPENDIX REFAPPDXCTSWE CAN WRITE 0 LEQ  XBF  ALPHA YBF 2   YBF2LEFT ALPHA  FRACLA    XBFYBFRA YBF2 ALPHABAR  FRACOVERLINELA    XBFYBFRA      YBF2RIGHT  FRACLA XBFYBFRA2 YBF2   XBF2THEN THE MINIMUM VALUE OF  XBFALPHA YBF2 IS OBTAINED WHEN ALPHA  FRACLA XBFYBFRAYBF2IN WHICH CASE THE COMPLETION OF THE SQUARE LEAVES  FRACLA XBFYBFRA2 YBF2   XBF2 GEQ 0FROM WHICH THE DESIRED INEQUALITY FOLLOWSNOW EXAMINE THE CONDITION FOR EQUALITY  IF YBFALPHA XBF THEN EQUALITYIN REFEQSW1 IS IMMEDIATE  ON THE OTHER HAND SUPPOSE THAT THEEQUALITY IN REFEQSW1 IS SATISFIED  THEN WORKING BACKWARDTHROUGH REFEQSW2 INDICATES THAT XBF  ALPHA YBF  0  BUT BYTHE PROPERTIES OF A NORM THIS MEANS THAT XBF  ALPHA YBF FOR SOMEALPHAENDPROOFTHIS THEOREM APPLIES TO EM ANY NORMED LINEAR VECTOR SPACE WITH ANINDUCED NORM  FOR THE VECTOR SPACE RBBN WITH THE EUCLIDEAN INNERPRODUCT THE CAUCHYSCHWARZ INEQUALITY IS BOXEDXBFT YBF2 LEQ XBFT XBFYBFT YBFFOR THE VECTOR SPACE CBBN WITH THE EUCLIDEAN INNERPRODUCT THE CAUCHYSCHWARZ INEQUALITY IS XBFH YBF2 LEQXBFH XBFYBFH YBF FOR THE VECTOR SPACE OF REAL FUNCTIONS DEFINED OVER AB THECAUCHYSCHWARZ INEQUALITY IS BOXEDLEFTINTAB FTGTDTRIGHT2 LEQ INTAB  F2TDT INTAB  G2TDTUSING THE CAUCHYSCHWARZ INEQUALITY WE CAN NOW SHOW THAT THE INDUCEDNORM SATISFIES THE REQUIRED TRIANGLE INEQUALITY PROPERTY  FOR VECTORSXBF AND YBF WHICH WE ASSUME FOR CONVENIENCE TO BE REAL WE HAVE BEGINALIGNED   XBF  YBF 2  LA XBFYBFXBFYBFRA  LA XBFXBFRA  2 LA XBFYBFRA  LA YBFYBF RA   LEQ LA XBFXBFRA  2 XBF   YBF  LA YBFYBF RA   XBF  YBF2ENDALIGNEDSECTIONDIRECTION OF VECTORS ORTHOGONALITYLABELSECDIRVECTHE INNER PRODUCT CAN BE USED TO DEFINE A DIRECTION OF ANGULARSEPARATION BETWEEN VECTORS AND HENCE A CONCEPT OF DIRECTIONFOR VECTORS XBF AND YBF IN RBB3 OR RBB2 IT IS WELLKNOWN THAT THE COSINE OF THE ANGLE BETWEEN THE VECTORS IS COS THETA  FRACLA XBFYBFRAXBF2 YBF2 NOTE THAT THE 2NORM  WHICH IS THE INDUCED NORM  IS USED INDEFINING THE LENGTH  USING THE CAUCHYSCHWARZ INEQUALITY IT CAN BESHOWN THATBEGINEQUATION 1 LEQ FRACLA XBFYBFRAXBF2 YBF2 LEQ 1LABELEQANGLEBOUNDENDEQUATIONSO THE ANGLE THETA IS REAL  THIS SAME EXPRESSION WITH THEAPPROPRIATE INNER PRODUCT DEFINES DIRECTION IN ANY INNER PRODUCT SPACEBEGINEXAMPLE  CONSIDER THE VECTORS XBF  1 2 3 4TQQUAD YBF  4 2 4 5TTHEN THE ANGLE THETA BETWEEN THE VECTORS IS DETERMINED BY COS THETA  FRACLA XBFYBFRAXBF YBF  0935ENDEXAMPLEBEGINEXAMPLE FOR FUNCTIONS DEFINED ON 01 FIND THE ANGLE  BETWEEN THE FUNCTIONS  X1T  1T2QQUADTEXTANDQQUADX2T  T22TFIRST COMPUTE  X1   LEFTINT01 X1T2DTRIGHT12 SQRT2815AND  X2   LEFTINT01 X2T2DTRIGHT12 SQRT815THEN COS THETA  FRACINT01 X1TX2TDTX1 X2 FRAC298SQRT14ENDEXAMPLEBEGINDEFINITION  IF XBF AND YBF ARE NONZERO VECTORS AND XBFALPHA YBF FOR  SOME SCALAR ALPHA THEN XBF AND YBF ARE SAID TO BE BF    COLINEAR  INDEXCOLINEAR IN AN INNERPRODUCT SPACE THIS  MEANS THAT THE ANGLE BETWEEN XBF AND YBF SATISFIES COS  THETA  PM 1ENDDEFINITIONA GEOMETRIC CONCEPT WHICH WILL BE OF CONSIDERABLE IMPORTANCE TO US ISTHE IDEA OF ORTHOGONAL VECTORSBEGINDEFINITION  VECTORS X AND Y IN AN INNER PRODUCT SPACE ARE SAID TO BE BF    ORTHOGONAL INDEXORTHOGONAL IF LA XY RA  0 NOTATIONALLY THIS IS DENOTED AS X PERP YINDEXPERPPERPSEEORTHOGONAL THE WORDS PERPENDICULARINDEXPERPENDICULARSEEORTHOGONAL AND NORMALINDEXNORMALSEEORTHOGONAL ARE SYNONYMOUS WITH ORTHOGONALENDDEFINITIONTHE ZERO VECTOR IS ORTHOGONAL TO EVERY OTHER VECTORBEGINDEFINITIONA SET OF VECTORS  PBF1PBF2LDOTSPBFM IS SAID TO BE BF  ORTHONORMAL INDEXORTHONORMAL IF THEY ARE MUTUALLY PAIRWISEORTHOGONAL AND EACH HAVE UNIT LENGTH LA PBFIPBFJ RA  DELTAIJWHERE DELTAIJ IS THE BF KRONECKER DELTA INDEXKRONECKER DELTA FUNCTION DEFINED BY INDEXDELTA FUNCTION DELTAIJ  BEGINCASES  1  I J  0  TEXTOTHERWISEENDCASESENDDEFINITIONFOR ORTHOGONAL VECTORS REGARDLESS OF THE INNER PRODUCT THE FAMILIARPYTHAGOREAN THEOREM HOLDSBEGINLEMMA LABELLEMPYTH THE PYTHAGOREAN THEOREM  INDEXPYTHAGOREAN THEOREM IF XBF PERP YBF AND  CDOT IS  AN INDUCED NORM THEN FOR THE NORM  CDOT  INDUCED FROM THE  INNER PRODUCTBEGINEQUATION XBF  YBF2  XBF2  YBF2LABELEQPYTHAG1ENDEQUATIONCONVERSELY IF REFEQPYTHAG1 HOLDS THEN XBF PERP YBFENDLEMMATHE  PROOF IS STRAIGHTFORWARDBEGINEXAMPLE  CONSIDER THE SET OF POLYNOMIALS  P0T1 QQUAD P1T  T QQUAD P2T FRAC123T21 QQUADP3T  FRAC125T3  3TP4T  FRAC1835T4  30T2  3THEN IT MAY BE VERIFIED BY DIRECT COMPUTATION THAT WHEN THE INNERPRODUCT IS DEFINED AS LA FG RA INT11 FTGTDTTHESE POLYNOMIALS ARE ORTHOGONAL LA PMPN RA  BEGINCASES  0  M NEQ N   FRAC22N1  M  NENDCASESTHESE POLYNOMIALS ARE THE FIRST FEW EM LEGENDRE POLYNOMIALS ALL OFWHICH ARE ORTHOGONAL OVER 11 INDEXLEGENDRE POLYNOMIALENDEXAMPLEGEOMETRIC INSIGHT CAN OFTEN BE OBTAINED BY DRAWING QUALITATIVECOORDINATE SYSTEMS THAT DEMONSTRATE SUBSPACES ORTHOGONALITY ETCWITHOUT NECESSARY REGARD TO THE DETAILS OF THE LENGTHS OF VECTORS ORTHE ANGLES BETWEEN VECTORS  BASED ON THE GEOMETRIC UNDERSTANDING SUCHCOORDINATE SYSTEMS AFFORD IT MAY BE EASIER TO PROVIDE MATHEMATICALSTATEMENTS FOR THE GEOMETRIC CONSTRUCTIONSBEGINEXAMPLE  LET X1T  1  X2T  T AND X3T  T2  FOR T IN 01  THEN LA X1X2 RA  0QUADQUAD LA X1X3 RA  13 QUADQUADLA X2X3 RA  14SO X1 AND X2 ARE ORTHOGONAL BUT THE OTHER PAIRS OF FUNCTIONSARE NOT  THIS MAY BE DIAGRAMMED AS SHOWN IN FIGURE REFFIGQG1WHERE THE ORTHOGONALITY HAS BEEN EXPLICITLY SHOWN BUT THE PARTICULARANGLES BETWEEN OTHER VECTORS HAS NOT BEENENDEXAMPLEBEGINEXERCISES  ITEM SHOW THAT FOR AN INDUCED NORM CDOT    BEGINEQUATION      LABELEQPARALLELOGRAM       XY 2   XY2  2X2 2Y2    ENDEQUATIONTHIS EQUATION IS KNOWN AS THE PARALLELOGRAM LAW  IN TWODIMENSIONALGEOMETRY AS SHOWN IN FIGURE REFFIGPARALLELOGRAM THE RESULT SAYSTHAT THE SUM OF  SQUARES OF THE LENGTHS OF THE DIAGONALS IS EQUAL TOTWICE THE SUM OF THE SQUARES OF THE ADJACENT SIDES A SORT OF TWOFOLDPYTHAGOREAN THEOREMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPARALLEL    CAPTIONTHE PARALLELOGRAM LAW    LABELFIGPARALLELOGRAM  ENDCENTERENDFIGUREITEM PROVE LEMMA REFLEMPYTHITEM SHOW THAT  LA XBFYBFRA  FRACXBF  YBF22  XBF  YBF224 THIS IS KNOWN AS THE POLARIZATION IDENTITYITEM PROVE REFEQPYTHAG1ITEM SHOW THAT THE INEQUALITY REFEQANGLEBOUND IS TRUE  ITEM LET X1T  3T2  1 X2T 5T3  3T AND X3T     2T2  T AND DEFINE THE INNER PRODUCT AS LA FGRA     INT11 FTGTDT  COMPUTE THE ANGLES EACH PAIRWISE    COMBINATION OF THESE FUNCTIONS AND IDENTIFY FUNCTIONS THAT ARE    ORTHOGONAL  ITEM LET  BEGINALIGNEDXBF1  1 2 4 2T XBF2  5231T XBF3  1212TENDALIGNEDAND COMPUTE THE ANGLES BETWEEN THESE VECTORS USING THE EUCLIDEANINNER PRODUCT AND IDENTIFY WHICH VECTORS ARE ORTHOGONALITEM SHOW THAT A SET OF VECTORS P1P2LDOTSPM WHICH ARE  MUTUALLY ORTHOGONAL SO THAT LA PIPJ RA  0 TEXT IF  I NEQ JIS LINEARLY INDEPENDENT  ORTHOGONALITY IMPLIES LINEAR INDEPENDENCEITEM LET S BE A VECTOR SPACE WITH AN INDUCED NORM  SHOW THAT  LA XBFYBF RA   XBF   YBFIF AND ONLY IF A XBF   B YBF  0 FOR SOME SCALARS A AND BENDEXERCISESSECTIONWEIGHTED INNER PRODUCTSLABELSECWIPFOR A FINITEDIMENSIONAL VECTOR SPACE A BF WEIGHTED INNER PRODUCTINDEXWEIGHTED INNER PRODUCTCAN BE OBTAINED BY INSERTING A HERMITIAN WEIGHTING MATRIX W BETWEENTHE ELEMENTS INNERPXBFYBFW  XBFH W YBF  YBFH W XBF THE CONCEPT OF ORTHOGONALITY IS DEFINED WITH RESPECT TO THE PARTICULARINNER PRODUCT USED CHANGING THE INNER PRODUCT MAY CHANGE THEORTHOGONALITY RELATIONSHIP BETWEEN VECTORSBEGINEXAMPLE  CONSIDER THE VECTORS XBF1  BEGINBMATRIX11 ENDBMATRIXQQUADQQUAD XBF2 BEGINBMATRIX2 1 ENDBMATRIXIT IS EASILY VERIFIED THAT THESE VECTORS ARE NOT ORTHOGONAL WITHRESPECT TO THE USUAL INNER PRODUCT XBF1TXBF2  HOWEVER FORTHE WEIGHTED INNER PRODUCT LA XBFYBFRAW  XBFTBEGINBMATRIX2  2  2  2ENDBMATRIX YBFTHE VECTORS XBF1 AND XBF2 ARE ORTHOGONALENDEXAMPLEIN ORDER FOR THE WEIGHTED INNER PRODUCT TO BE USED TO DEFINE A NORMAS IN  XBF W2  INNERPXBFXBFW  XBFH W XBFIT IS NECESSARY THAT XBFH W XBF  0 FOR ALL XBF NEQ 0  AMATRIX W WITH THIS PROPERTY IS SAID TO BE BF POSITIVE DEFINITEINDEXPOSITIVE DEFINITEBEGINEXAMPLE  THE WEIGHTED INNER PRODUCT OF THE PREVIOUS EXAMPLE CANNOT BE USED AS  A NORM BECAUSE FOR ANY VECTOR OF THE FORM XBF  BEGINBMATRIX ALPHA  ALPHA ENDBMATRIXTHE PRODUCT XBFT W XBF  0 WHICH VIOLATES THE CONDITIONS FOR A NORMENDEXAMPLEWEIGHTING CAN ALSO BE APPLIED TO INTEGRAL INNER PRODUCTS  IF THERE ISSOME FUNCTION WT GEQ 0 OVER AB THEN AN INNER PRODUCT CAN BEDEFINED AS LA FGRAW  INTAB WT FT GT DTTHE WEIGHTING CAN BE USED TO PLACE MORE EMPHASIS ON CERTAIN PARTS OFTHE FUNCTION  MORE PRECISELY WE MUST HAVE WT GEQ 0 WITHWT0 ONLY ON A SET OF MEASURE ZEROBEGINEXAMPLE   LABELEXMCHEBYPOL LET US DEFINE A SET OF POLYNOMIALS BY TNT  COSN COS1TFOR T IN 11  THE FIRST FEW OF THESE OBTAINED BY APPLICATIONOF TRIGONOMETRIC IDENTITIES ARE T0T 1 QQUAD T1T  T QQUAD T2T  2T21 QQUAD T3T 4T3  3TA PLOT OF THE FIRST FEW OF THESE IS SHOWN IN FIGURE REFFIGCHEBPOLYINDEXCHEBYSHEV POLYNOMIAL INDEXORTHOGONAL POLYNOMIALTHESE POLYNOMIALS ARE THE EM CHEBYSHEV POLYNOMIALS  THEY HAVE THEINTERESTING PROPERTY THAT OVER THE INTERVAL 11 ALL THE EXTREMAOF THE FUNCTIONS HAVE THE VALUES 1 OR 1  THIS PROPERTY MAKES THEMVERY USEFUL FOR APPROXIMATION OF FUNCTIONS AS DISCUSSED IN CHAPTERREFCHAPPROXFURTHERMORE THE CHEBYSHEV POLYNOMIALS AREORTHOGONAL WITH WEIGHT FUNCTION WT  FRAC1SQRT1T2OVER THE INTERVAL 11  THE ORTHOGONALITY RELATIONSHIP BETWEENTHE CHEBYSHEV POLYNOMIALS IS INT11 FRAC1SQRT1T2 TNT TMT  DT  PIDELTANMENDEXAMPLEBEGINFIGUREHTBP  CENTERLINEEPSFIGFILEPICTUREDIRCHEBY1EPS CHEBYPLOTM  CAPTIONCHEBYSHEV POLYNOMIALS PROTECTTPROTECT0TPROTECT    THROUGH PROTECTTPROTECT5TPROTECT FOR T IN     11   LABELFIGCHEBPOLYENDFIGUREWE CAN DEFINE A WEIGHTED INNER PRODUCT ON THE VECTOR SPACE OFMATSIZEMN MATRICESBY LA AB RA  TRACEAH W BWHERE W IS A SYMMETRIC POSITIVEDEFINITE MATSIZEMM MATRIXUSING A NORM INDUCED FROM A WEIGHTED INNER PRODUCT WE CAN DEFINE AWEIGHTED DISTANCE BETWEEN TWO VECTORSBEGINEQUATION DWXBFYBF2   XBF  YBFW2  XBFYBFH WXBFYBFLABELEQMAHAL1ENDEQUATIONBEGINEXAMPLE  A WEIGHTED DISTANCE ARISES NATURALLY IN MANY SIGNAL DETECTION  ESTIMATION AND PATTERN RECOGNITION PROBLEMS IN NONWHITE GAUSSIAN  NOISE  IN THIS EXAMPLE A DETECTION PROBLEM IS CONSIDERED  DETECTION PROBLEMS ARE DISCUSSED MORE FULLY IN CHAPTER  REFCHAPDETECTION    LET SBF IN RBBN BE A SIGNAL WHICH TAKES ON ONE OF TWO  DIFFERENT VALUES EITHER SBF  SBF0 OR SBF  SBF1  ONE OF  THESE SIGNALS IS CHOSEN AT RANDOM WITH EQUAL PROBABILITY  EITHER  BY A BINARY DATA TRANSMITTER OR BY NATURE  THE SIGNAL SBF IS  OBSERVED IN THE PRESENCE OF ADDITIVE GAUSSIAN NOISE NBF WHICH HAS  MEAN ZEROBF AND COVARIANCE MATRIX R  THE OBSERVATION YBF  CAN BE MODELED AS YBF  SBF  NBF FROM THE OBSERVATION OF YBFYBF WE DESIRE TO DETERMINE WHICH VALUEOF SBF ACTUALLY OCCURRED  THIS IS THE BF DETECTION PROBLEMCONDITIONED UPON A VALUE OF SBFSBF THE OBSERVATION IS GAUSSIAN WITHMEAN SBF AND THE SAME COVARIANCE FYBFSBF  SBF  FRAC12PIN2DETR12EXPFRAC12 YBFSBFT R1 YBFSBFWHERE EITHER SBFSBF0 OR SBF  SBF1  FROM THE OBSERVATIONYBF WE CAN COMPUTE THE EM LIKELIHOOD THAT THE SIGNAL WASPRODUCED BY SBF FOR EACH OF THE POSSIBLE VALUES OF SBF THENSELECT THE ONE WITH THE HIGHEST LIKELIHOOD  THAT IS WE COMPAREBEGINEQUATION FYBFSBFSBF0QQUAD TEXTWITHQQUADFYBFSBFSBF1LABELEQDETECT1ENDEQUATIONAND DETERMINE OUR DECISION ABOUT SBF ON THE BASIS OF WHICHLIKELIHOOD FUNCTION IS LARGEST  THIS IS THE MAXIMUM LIKELIHOODDECISION RULE  CANCELING COMMON FACTORS IN THE COMPARISON THIS ISEQUIVALENT TO COMPARINGBEGINEQUATION YBFSBF0T R1 YBFSBF0QQUAD TEXTWITHQQUADYBFSBF1T R1YBFSBF1LABELEQDETECT2ENDEQUATIONAND CHOOSING EITHER SBF0 OR SBF1 DEPENDING UPON WHICHQUANTITY IS SMALLER  THESE QUANTITIES CAN BE OBSERVED TO BE WEIGHTEDDISTANCES OF THE FORM REFEQMAHAL1  LET W  R1 AND DEFINETHE WEIGHTED INNER PRODUCT IN RBBN BY LA XBFYBFRAW  XBFT W YBFTHIS INDUCES A WEIGHTED NORM  XBFW2  XBFT W XBFTHE COMPARISON IN REFEQDETECT2 CORRESPONDS TO COMPUTING  YBF  SBF0W QQUAD TEXTAND QQUAD    YBF  SBF1WWITH THE MAXIMUM LIKELIHOOD CHOICE BEING THAT WHICH HAS THE MINIMUMWEIGHT DISTANCE  THIS WEIGHED DISTANCE MEASURE ARISES COMMONLY INPATTERN RECOGNITION PROBLEMS AND IS KNOWN AS THE EM MAHALONOBIS  DISTANCE INDEXPATTERN RECOGNITION INDEXMAHALONOBIS DISTANCEFURTHER SIMPLIFICATIONS ARE OFTEN POSSIBLE IN THIS COMPARISONBEGINALIGNYBFSBF0W  YBFT W YBF  YBFT W SBF0  SBF0T W YBF SBF0T W SBF0  YBFT W YBF  2YBFT W SBF0  SBF0T W SBF0ENDALIGNAND SIMILARLY FOR YBF  SBF1W  IF SBF0 AND SBF1HAVE THE SAME INNER PRODUCT NORM SO SBF0T W SBF0  SBF1T WSBF1 THEN WHEN COMPARING YBFSBF0W WITHYBFSBF1W THESE TERMS CANCEL AS WELL AS THE YBFTWYBFTERM  THE CHOICE IS MADE DEPENDING ON WHETHER YBFT WSBF0 QQUAD TEXTORQQUAD YBFT W SBF1 IS LARGER THAT IS DEPENDING ON WHICH WEIGHTED INNER PRODUCT ISLARGEST  THE INNER PRODUCT IS THUS SEEN TO BE A SIMILARITY MEASURETHE SIGNAL SBF IS CHOSEN THAT IS MOST SIMILAR TO THE RECEIVEDSIGNAL VECTOR WHERE THE SIMILARITY IS DETERMINED BY THE WEIGHTEDINNER PRODUCTENDEXAMPLESUBSECTIONEXPECTATION AS AN INNER PRODUCTTHE EXAMPLES OF WEIGHTED INNER PRODUCTS UP UNTIL NOW HAVE BEEN OFDETERMINISTIC FUNCTIONS  AN IMPORTANT GENERALIZATION DEVELOPS WHEN AA JOINT DENSITY IS USED AS A WEIGHTING FUNCTION IN THE INNER PRODUCTLET X AND Y BE RANDOM VARIABLES WITH JOINT DENSITY FXYXYWE DEFINE AN INNER PRODUCT BETWEEN THEM AS LA XYRA  INT X Y FXYXY  DXDYTHIS INNER PRODUCT IS OF COURSE AN EXPECTATION AND INTRODUCTION OFTHIS INNER PRODUCT ALLOWS THE CONCEPTUAL POWER OF VECTOR SPACES TO BEAPPLIED TO MEANSQUARE ESTIMATION THEORY  THUS LA XY RA  EXYE IS THE EXPECTATION OPERATOR  ORTHOGONALITY IS DEFINED FORRANDOM VARIABLES AS IT IS FOR DETERMINISTIC QUANTITIES THE RANDOMVARIABLES X AND Y ARE ORTHOGONAL IF EXY  0 THE INNER PRODUCTINDUCES A NORM LA XX RA  E X2IF X IS A ZEROMEAN RV THEN LA XXRA  VARX IS AN INDUCEDNORMFOOTNOTEAS WITH OTHER FUNCTION SPACES THERE ARE SOME TECHNICAL  PROBLEMS ASSOCIATED WITH VECTOR SPACES OVER PROBABILITY SPACES  SINCE THERE MAY BE RANDOM VARIABLES X AND Y SUCH THAT  XY    0 BUT X NEQ Y ALWAYS  HOWEVER IT CAN BE SHOWN THAT IF   XY  0 THEN XY AS ALMOST SURELY THAT IS EXCEPT ON A  SET OF PROBABILITY MEASURE 0  WE CAN ALSO DEFINE AN INNER PRODUCT BETWEEN RANDOM EM VECTORS  LETXBF  X1X2LDOTSXNT AND YBF  Y1Y2LDOTSYNT BENDIMENSIONAL RANDOM VECTORS  THEN WE CAN DEFINE AN INNER PRODUCTBETWEEN THESE VECTORS AS LA XBF YBF RA  E SUMI1N XI YBARINOTE THAT WE CAN WRITE THIS INNER PRODUCT AS LA YBF YBF RA    EYBFH YBFANOTHER NOTATION THAT IS SOMETIMES CONVENIENT IS TO WRITE LA YBF YBF RA  TRACE EYBF YBFHWHERE THE TRACEX IS THE TRACE OPERATOR INDEXTRACE THE SUMOF THE ELEMENTS ON THE DIAGONAL OF THE SQUARE MATRIX X  SEESECTION REFSECTPOSETRACEWHEN THE VECTORSPACE VIEWPOINT IS APPLIED TO PROBLEMS OFMINIMIZATION AS DISCUSSED SUBSEQUENTLY THERE ARE TWO MAJORAPPROACHES TO THE PROBLEM  IN THE FIRST CASE AN INNER PRODUCT ISUSED THAT IS NOT BASED ON AN EXPECTATION MINIMIZATION OF THIS SORTIS REFERRED TO AS EM LEASTSQUARES LS INDEXLEASTSQUARES INTHE SIGNAL PROCESSING LITERATURE  WHEN AN INNER PRODUCT IS USED THATIS DEFINED AS AN EXPECTATION THEN THE APPROXIMATION OBTAINED ISREFERRED TO AS A EM MINIMUM MEANSQUARES MMS APPROXIMATIONINDEXMINIMUM MEANSQUARE  IN FACT BOTH APPROXIMATION TECHNIQUESRELY ON PRECISELY THE SAME THEORY BUT SIMPLY EMPLOY INNER PRODUCTSSUITED TO THE NEEDS OF THE PARTICULAR PROBLEMBEGINEXERCISESITEM PERFORM THE SIMPLIFICATIONS TO GO FROM THE COMPARISON IN    REFEQDETECT1 TO THE COMPARISON IN REFEQDETECT2  ITEM SHOW BY INTEGRATION THAT INT11 FRAC1SQRT1T2 TNT TMT  BEGINCASES  PI  NM0   PI2  NM NNEQ 0   0  N NEQ MENDCASESHINT USE T  COS X IN THE INTEGRALENDEXERCISESSECTIONHILBERT AND BANACH SPACESLABELSECHILBERTWITH THE DEFINITIONS OF METRIC SPACES AND INNERPRODUCT SPACES BEHINDUS WE ARE NOW READY TO INTRODUCE THE SPACES IN WHICH MOST OF THE WORKIN SIGNAL PROCESSING IS PERFORMEDBEGINDEFINITION  A COMPLETE NORMED VECTOR SPACE IS CALLED A BF BANACH SPACE  INDEXBANACH SPACE A COMPLETE NORMED VECTOR SPACE WITH AN INNER  PRODUCT IN WHICH THE NORM IS THE INDUCED NORM IS CALLED A BF    HILBERT SPACE INDEXHILBERT SPACEENDDEFINITIONSOME EXAMPLES OF BANACH AND HILBERT SPACESBEGINENUMERATEITEM THE SPACE OF CONTINUOUS FUNCTIONS CABDINFTY FORMS A  BANACH SPACE  RECALL THAT IN EXAMPLE REFEXMXINF  C11DINFTY WAS SHOWN TO BE COMPLETEITEM HOWEVER THE SPACE OF FUNCTIONS CAB WITH THE LP NORM P   INFTY DOES EM NOT FORM A BANACH SPACE SINCE IT IS NOT  COMPLETE  WE SAW IN EXAMPLE REFEXMFNSEQ A SEQUENCE OF  CONTINUOUS FUNCTIONS THAT DOES NOT HAVE A LIMIT IN C11ITEM THE SEQUENCE SPACE LP0INFTY IS A BANACH SPACE  WHEN  P2 IT IS A HILBERT SPACEITEM THE SPACE LPAB IS A BANACH SPACE  WHEN P2 IT IS A  HILBERT SPACE  THE HILBERT SPACE OF FUNCTIONS WITH DOMAIN OVER THE  WHOLE REAL LINE IS DENOTED LPRBBENDENUMERATEBECAUSE OF THE UTILITY OF HAVING THE NORM INDUCED FROM AN INNERPRODUCT THE EMPHASIS IN THIS AND SUCCEEDING CHAPTERS IS ON HILBERTSPACES  INPUTLINALGDIRHILBERTBOXTEXIT CAN BE SHOWN CITEP 267NAYLORSELL THAT IF A NORMED VECTORSPACE IS FINITE DIMENSIONAL THEN IT IS COMPLETE  HENCE EVERY NORMEDFINITE DIMENSIONAL SPACE IS A BANACH SPACE IF THE NORM IS INDUCEDFROM AN INNER PRODUCT THEN IT IS ALSO A HILBERT SPACE  FURTHERMORE EVERY FINITEDIMENSIONAL SUBSPACE OF A SPACE ISCOMPLETESECTIONORTHOGONAL SUBSPACESLABELSECORTHOSUBBEGINDEFINITION  LET S BE A VECTOR SPACE AND LET V AND W BE SUBSPACES OF S  V AND W ARE BF ORTHOGONAL IF EVERY VECTOR VBF IN V IS  ORTHOGONAL TO EVERY VECTOR WBF IN WMC LA VBFWBF RA   0 INDEXORTHOGONAL SUBSPACEENDDEFINITIONBEGINDEFINITION  FOR A SUBSPACE V OF AN INNER PRODUCT SPACE S THE SPACE OF ALL  VECTORS ORTHOGONAL TO V IS CALLED THE BF ORTHOGONAL COMPLEMENT  OF V  THIS IS DENOTED AS VPERPENDDEFINITIONBEGINEXAMPLELET V BE THE PLANE SHOWN IN FIGURE REFFIGORTHOG1  THEN THEORTHOGONAL SPACE WVPERP IS SPANNED BY THE VECTOR WBF  ENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRORTHOGSPACE1    CAPTIONA SPACE AND ITS ORTHOGONAL COMPLEMENT    LABELFIGORTHOG1  ENDCENTERENDFIGURETHE ORTHOGONAL COMPLEMENT OF A SUBSPACE IS ITSELF A SUBSPACE SEEEXERCISE REFEXORTHOGCOMP1  THE ORTHOGONAL COMPLEMENT HAS THEFOLLOWING PROPERTIES SEE LUENBERGER P 52 NS P 294295BEGINTHEOREM CITELUENBERGER1969NAYLORSELL  LABELTHMORTHOGCOMP LET V  AND W BE SUBSETS OF AN INNER PRODUCT SPACE S NOT NECESSARILY  COMPLETE  THEN  BEGINENUMERATE  ITEM VPERP IS A CLOSED SUBSPACE OF S  ITEM V SUBSET VPERPPERP  ITEM IF V SUBSET W THEN WPERP SUBSET VPERP  ITEM VPERPPERPPERP  VPERP  ITEM IF X IN V CAP VPERP THEN X  0  ITEM VPERPPERP IS THE SMALLEST CLOSED SUBSPACE CONTAINING    S  THAT IS VPERPPERP  CLOSUREV  ENDENUMERATEENDTHEOREMBEGINPROOF  WE WILL PROVE PART 1  THE REST OF THE PROPERTIES ARE TO BE PROVED  AS AN EXERCISE SEE EXERCISE  REFEXORTHOCOMP  TO SHOW CLOSURE  OF VPERP LET  XBFN BE A CONVERGENT SEQUENCE IN  VPERP SO THAT XBFN RIGHTARROW XBF  THEN BY THE  CONTINUITY OF THE INNER PRODUCT SHOWN IN LEMMA REFLEMCONTIP WE  HAVE FOR ANY V IN V 0  LA XBFN VBFRA RIGHTARROW LA XBFVBFRASO THAT XBF IN VPERPENDPROOFWHAT IS PERHAPS A LITTLE SURPRISING AT FIRST ABOUT THIS THEOREM IS THEFACT THAT IT MAY EM NOT BE THE CASE THAT VPERPPERP  V  WHAT IS LACKING IS THE COMPLETENESS VPERPPERP MAY HAVE CAUCHYSEQUENCES IN IT THAT V DOES NOTBEGINEXERCISES    ITEM LABELEXORTHOGCOMP1 SHOW THAT THE ORTHOGONAL COMPLEMENT OF A SUBSPACE IS A SUBSPACEITEM LABELEXORTHOCOMP PROVE ITEMS 2 THROUGH 6 OF THEOREM  REFTHMORTHOGCOMPENDEXERCISESSECTIONLINEAR TRANSFORMATIONS RANGE AND NULLSPACELABELSECLINTRANSWE PAUSE IN OUR DEVELOPMENT OF VECTOR SPACES TO REINTRODUCE A CONCEPTTHAT SHOULD BE FAMILIARBEGINDEFINITIONINDEXTRANSFORMATIONLINEAR  A TRANSFORMATION LMC X RIGHTARROW Y FROM A VECTOR SPACE X TO A  VECTOR SPACE Y WHERE X AND Y HAVE THE SAME SCALAR FIELD R  IS A BF LINEAR TRANSFORMATION IF FOR ALL VECTORS X X1 X2 IN  X   BEGINENUMERATE  ITEM LALPHA X  ALPHA LX FOR ALL XBF IN X AND ALL    SCALARS ALPHA IN R AND  ITEM LX1  X2  LX1  LX2  ENDENUMERATEENDDEFINITIONWE WILL THINK OF LINEAR TRANSFORMATIONS AS EM OPERATORS INDEXOPERATORBEGINEXAMPLE WE WILL BEGIN WITH SEVERAL EXAMPLES FROM VECTOR SPACES  OF FUNCTIONS BEGINENUMERATEITEM LET X BE THE SET OF CONTINUOUS REALVALUED FUNCTIONS AND  DEFINE LMC X RIGHTARROW X BY LXT  INT0T HTAU XTTAUDTAUFOR ALL XT IN X  THEN L IS A LINEAR TRANSFORMATION WHICHCONVOLVES THE SIGNAL X WITH THE SIGNAL H INDEXCONVOLUTIONITEM LET X BE THE SET OF CONTINUOUS REALVALUED FUNCTIONS DEFINED  ON 01  THEN LMC X RIGHTARROW RBB DEFINED BY LXT  INT01 HTAU XTAUDTAUIS A LINEAR TRANSFORMATION AN INNER PRODUCTITEM LET X BE THE SET OF CONTINUOUS REALVALUED FUNCTIONS AND LET  TT0MC X RIGHTARROW X BE DEFINED BY TT0XT  BEGINCASES  XT  T  T0  0  TEXTOTHERWISEENDCASESWHERE T0 IS A PARAMETER OF THE TRANSFORMATION  THEN TT0 IS ALINEAR TRANSFORMATION  THIS TRANSFORMATION TRUNCATES A SIGNAL INTIME INDEXTRUNCATIONIN TIMEITEM LET X BE THE SET OF ALL FOURIER TRANSFORMABLE FUNCTIONS AND  LET Y BE THE SET OF FOURIER TRANSFORMS OF ELEMENTS IN X  DEFINE  FMC X RIGHTARROW Y BY FXT  INTINFTYINFTY XT EJOMEGA T DTTHE OPERATOR F IS A LINEAR OPERATORITEM LET BMC X RIGHTARROW X BE DEFINED BY BB0XT  FC1 TB0 XOMEGAWHERE XOMEGA IS THE FOURIER TRANSFORM OF XT FC1 ISTHE INVERSE FOURIER TRANSFORM OPERATOR AND TB0XOMEGATRUNCATES THE FOURIER TRANSFORM  THUS BB0XT IS A BANDLIMITEDSIGNAL INDEXTRUNCATIONIN FREQUENCY INDEXBANDLIMITED SIGNALENDENUMERATEENDEXAMPLEBEGINEXAMPLE PERHAPS MORE COMMONLY WE SEE LINEAR TRANSFORMATIONS  BETWEEN VECTOR SPACES OF FINITE DIMENSION  IN GENERAL A LINEAR  TRANSFORMATION L FROM THE VECTOR SPACE RN TO RM CAN BE  EXPRESSED USING THE NOTATION OF AN MATSIZEMN MATRIX L  THAT IS THE MATRIX BECOMES THE LINEAR TRANSFORMATION  BEGINENUMERATE  ITEM LET LMC RBB3 RIGHTARROW RBB2 BE DEFINED BY LX1X2X3  X1  2X2 3X2  4X3THIS LINEAR TRANSFORMATION CAN BE PLACED IN MATRIX NOTATION  BYWRITING AN ELEMENT IN RBB3 IN VECTOR FORM AS X1X2X3T INRBB3 WE CAN WRITE L  BEGINBMATRIX 120 034 ENDBMATRIXTHEN L XBF  BEGINBMATRIX X12X2  3X2  4X3 ENDBMATRIXITEM LET LMC RBB3RIGHTARROW RBB3 BE DEFINED BY THE MATRIX L  BEGINBMATRIX 001  010  100ENDBMATRIXTHEN L IS THE LINEAR TRANSFORMATION THAT REVERSES THE COORDINATES OFA VECTOR XBF IN RBB3  ENDENUMERATEENDEXAMPLECONSIDERABLY MORE IS SAID ABOUT LINEAR TRANSFORMATIONS BETWEENFINITEDIMENSIONAL VECTORS SPACES IN CHAPTER REFCHAPMATINVASSOCIATED WITH ANY OPERATOR LINEAR OR OTHERWISE ARE TWO IMPORTANTSPACES  THESE SPACES ARE THE RANGE AND THE NULLSPACE  TWO MORESPACES ASSOCIATED WITH LINEAR OPERATORS ARE PRESENTED IN SECTIONREFSEC4SUBOPBEGINDEFINITION  LET LMC XRIGHTARROW Y BE AN OPERATOR LINEAR OR OTHERWISE  THE BF RANGE SPACE INDEXRANGE RANGEL IS RANGEL  YBF  LXBFMC  XBF IN XTHAT IS IT IS THE SET OF VALUES IN Y THAT ARE REACHED FROM X BYAPPLICATION OF L  THE BF NULLSPACE INDEXNULLSPACE NULLSPACEL IS NULLSPACEL  XBF IN X LXBF  ZEROBFTHAT IS IT IS THE SET OF VALUES IN XBF THAT ARE TRANSFORMED TO ZEROBFIN Y BY L  THE NULLSPACE OF AN OPERATOR IS ALSO CALLED THE BF  KERNEL OF THE OPERATOR INDEXKERNELENDDEFINITIONLET A BE AN MATSIZENM MATRIX A  PBF1PBF2LDOTSPBFMWHICH WE REGARD AS A LINEAR OPERATOR  THEN A POINT XBF IN RBBMIS TRANSFORMED AS A XBF  X1 PBF1  X2 PBF2  CDOTS  XM PBFMWHICH IS A LINEAR COMBINATION OF THE COLUMNS OF A  THUS THE RANGEMAY BE EXPRESSED AS RANGEA  LSPANPBF1PBF2LDOTSPBFMTHE RANGE OF A MATRIX IS ALSO REFERRED TO AS THE EM COLUMN SPACEINDEXCOLUMN SPACESEERANGE OF A  THE NULLSPACE IS THAT SET OFVECTORS SUCH THAT AXBF  ZEROBFBEGINEXAMPLE  LET  A  BEGINBMATRIX 1  0  0 0  0  0 1  0  1 ENDBMATRIXTHEN THE RANGE OF A IS LSPAN101T 001TTHE NULLSPACE OF A IS  NULLSPACEA  LSPAN010TENDEXAMPLEBEGINEXAMPLE BEGINENUMERATEITEM LET LXT  INT0T XTAU HTTAUDTAU  THEN THE  NULLSPACE OF L IS THE SET OF ALL FUNCTIONS XT THAT RESULT IN  ZERO WHEN CONVOLVED WITH HT  FROM SYSTEMS THEORY WE REALIZE  THAT WE CAN TRANSFORM THE CONVOLUTION OPERATION AND MULTIPLY IN THE  FREQUENCY DOMAIN  FROM THIS PERSPECTIVE WE PERCEIVE THAT THE  NULLSPACE OF L IS THE SET OF FUNCTIONS WHOSE FOURIER TRANSFORMS DO  NOT SHARE ANY SUPPORT WITH THE SUPPORT INDEXSUPPORT OF THE  FOURIER TRANSFORM OF HITEM LET LXT  INT0T XTAU HTAU DTAU WHERE X IS THE  SET OF CONTINUOUS FUNCTIONS  THEN RANGEL IS THE SET OF REAL  NUMBERS UNLESS HT EQUIV 0ITEM THE RANGE OF THE OPERATOR A  BEGINBMATRIX 10 0 0 ENDBMATRIXIS THE SET OF ALL VECTORS OF THE FORM C0T  THE NULLSPACE OFTHIS OPERATOR IS LSPAN01ENDENUMERATEENDEXAMPLEBEGINEXERCISESITEM   LET X AND Y BE VECTOR SPACES OVER THE SAME SET OF  SCALARS  LET LTXY DENOTE THE SET OF ALL LINEAR TRANSFORMATIONS  FROM X TO Y  LET L AND M BE OPERATORS FROM LTXY  DEFINE AN ADDITION OPERATOR BETWEEN L AND M AS LMX  LX MXFOR ALL X IN X  ALSO DEFINE SCALAR MULTIPLICATION BY ALX  ALXSHOW THAT LTXYIS A LINEAR VECTOR SPACEITEM LET X Y AND Z BE LINEAR VECTOR SPACES OVER THE SAME SET  OF SCALARS AND LET L1MC XRIGHTARROW Y AND L2MC Y  RIGHTARROW Z BE LINEAR OPERATORS SHOW THAT THE COMPOSITION  L2L1MC XRIGHTARROW Z IS A LINEAR OPERATORENDEXERCISESSECTIONINNERSUM AND DIRECTSUM SPACESLABELSECISDSPBEGINDEFINITION  IF V AND W ARE LINEAR SUBSPACES THE SPACE V  W IS THE BF    INNER SUM INDEXINNER SUM SPACE CONSISTING OF ALL  POINTS XBF  VBF  WBF WHERE VBF IN V AND WBF IN WENDDEFINITIONBEGINEXAMPLE LABELEXMVS1  CONSIDER S  GF23 INDEXGF2GF2 THAT IS THE SET OF  ALL 3TUPLES OF ELEMENTS OF GF2 SEE BOX REFBOXGF2  THEN  FOR EXAMPLE XBF  101 IN SQQUAD TEXTANDQQUAD YBF  001IN SAND XBF  YBF  100LET W  LSPAN010 AND V LSPAN100 BE TWO SUBSPACES IN S  THEN W  000010AND V  000100THESE TWO SUBSPACES ARE  ORTHOGONALTHE ORTHOGONAL COMPLEMENT TO V IS  VPERP  000010001011THUS W SUBSET VPERPTHE INNER SUM SPACE OF V AND W IS VW  000010100110ENDEXAMPLEBEGINDEFINITION  TWO LINEAR SUBSPACES V AND W OF THE SAME DIMENSIONALITY ARE BF    DISJOINT INDEXDISJOINT IF V CAP W   0   THAT IS THE  ONLY VECTOR THEY HAVE IN COMMON IS THE ZERO VECTOR  DISJOINT  SUBSPACES ARE SLIGHTLY DIFFERENT FROM DISJOINT SETS SINCE DISJOINT  SUBSPACES MUST HAVE THE ZERO VECTOR IN COMMON WHEREAS DISJOINT SETS  HAVE NO ELEMENTS IN COMMONENDDEFINITIONBEGINEXAMPLE  IN FIGURE REFFIGDISJOINT1 THE PLANE S IS A VECTOR SPACE IN  TWO DIMENSIONS AND V AND W ARE TWO ONEDIMENSIONAL SUBSPACES  INDICATED BY THE LINES IN THE FIGURE  THE ONLY POINT THEY HAVE IN  COMMON IS THE ORIGIN SO THEY ARE DISJOINT  NOTE THAT THEY ARE NOT  NECESSARILY ORTHOGONALENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRDISJOINT1    CAPTIONDISJOINT LINES IN RBB2    LABELFIGDISJOINT1  ENDCENTERENDFIGUREWHEN S  VW AND V AND W ARE DISJOINT W IS SAID TO BE THEEM ALGEBRAIC COMPLEMENT OF V  INDEXALGEBRAIC COMPLEMENT THELAST EXAMPLE ILLUSTRATES AN ALGEBRAIC COMPLEMENT THE INNER SUM OF THETWO LINES GIVES THE ENTIRE VECTOR SPACE S  ON THE OTHER HAND THESETS V AND W IN EXAMPLE REFEXMVS1 ARE NOT ALGEBRAICCOMPLEMENTS SINCE V  W IS NOT THE SAME AS S  AN ALGEBRAICCOMPLEMENT TO THE SET V OF THAT EXAMPLE WOULD BE THE SET Z  LSPAN010001 000010001011IT IS STRAIGHTFORWARD TO SHOW THAT IN ANY VECTOR SPACE S  EVERYLINEAR SUBSPACE HAS AN ALGEBRAIC COMPLEMENT  LET B BE A HAMELINDEXHAMEL BASISBASIS FOR S AND LET B1 SUBSET B BE A HAMEL BASIS FOR VTHEN LET B2  B  B1 THE SET DIFFERENCE SO THAT B1 CAP B2 EMPTYSET  THEN W  LSPANB2IS A HAMEL BASIS FOR THE ALGEBRAIC COMPLEMENT OF VTHE DIRECT SUM OF DISJOINT SPACES CAN BE USED TO PROVIDE A UNIQUEREPRESENTATION OF A VECTORBEGINLEMMA LABELLEMVWUNIQUE CITENAYLORSELL  LET V AND W BE  SUBSPACES OF A VECTOR SPACE S  THEN FOR  EACH XBF IN VW THERE IS A EM UNIQUE VBF IN V AND A EM    UNIQUE WBF IN W SUCH THAT XBF  VBF  WBF IF AND ONLY IF  V AND W ARE DISJOINTENDLEMMABEGINPROOFASSUME THAT V AND W ARE DISJOINT  THEN IF THERE ARE TWOREPRESENTATIONS FOR XBF XBF  VBF1  WBF1  VBF2  WBF2THEN VBF1 VBF2  WBF1  WBF2  BUT SINCE VBF1VBF2 INV AND WBF1WBF2 IN W AND V CAP W  0 WE MUST HAVE VBF1VBF2  0 AND WBF1  WBF2  0CONVERSELY SUPPOSE THAT THERE IS A UNIQUE REPRESENTATION XBF  VBF WBF FOR EACH XBF IN VW  ASSUME AS A CONTRADICTION THAT VAND W ARE NOT DISJOINT SO THAT THERE IS A NONZERO ELEMENT ZBF IN VCAP W  THEN WE CAN WRITE XBF  VBF  C ZBF  WBF  C ZBFWHERE C IS ANY SCALAR VALUE  BUT THIS LEADS TO A NONUNIQUEREPRESENTATION ENDPROOFANOTHER WAY OF COMBINING VECTOR SPACES IS BY THE DIRECT SUM  BEGINDEFINITION  THE BF DIRECT SUM INDEXDIRECT SUM OF LINEAR SPACES V AND  W DENOTED V OPLUS W IS DEFINED ON THE CARTESIAN PRODUCT  INDEXCARTESIAN PRODUCT V TIMES W SO A POINT IN VOPLUS W IS  AN ORDERED PAIR VW WITH V IN V AND W IN W  ADDITION IS  DEFINED COMPONENTWISE V1W1  V2W2  V1V2W1W2  SCALAR MULTIPLICATION IS DEFINED AS ALPHAVW  ALPHA VALPHA  WENDDEFINITIONTHE SUM VW AND THE DIRECT SUM VOPLUS W ARE DIFFERENT LINEARSPACES  HOWEVER IF V AND W ARE EM DISJOINT THEN VW AND VOPLUSW HAVE EXACTLY THE SAME STRUCTURE MATHEMATICALLY THEY ARE SIMPLYDIFFERENT REPRESENTATIONS OF THE SAME THING   WHEN DIFFERENTMATHEMATICAL OBJECTS BEHAVE THE SAME ONLY VARYING IN THE NAME THEOBJECTS ARE SAID TO BE EM ISOMORPHIC SEE BOX REFBOXISOMORPHBEGINTEXTBOX09TEXTWIDTHISOMORPHISMLABELBOXISOMORPHBEGINQUOTESOURCEWILLIAM SHAKESPEAREWHATS IN A NAME THAT WHICH WE CALL A ROSE BY ANY OTHER NAME WOULD SMELL AS SWEETENDQUOTESOURCEISOMORPHISM DENOTES THE FACT THAT TWO OBJECTS MAY HAVE THE SAMEOPERATIONAL BEHAVIOR EVEN IF THEY HAVE DIFFERENT NAMES INDEXISOMORPHISMAS AN EXAMPLE CONSIDER THE FOLLOWING TWO OPERATIONS FOR TWO GROUPSCALLED LA G1RA AND LA G2RABEGINCENTER    BEGINTABULARCCCCC    00011011 HLINE0000011011 0101001110 1010110001 1111100100    ENDTABULARQQUADQQUAD  BEGINTABULARCCCCCABCD HLINEAABCD BBADC CCDAB DDCBA  ENDTABULARENDCENTERCAREFUL COMPARISON OF THESE ADDITION TABLES REVEALS THAT THE SAMEOPERATION OCCURS IN BOTH TABLES BUT THE NAMES OF THE ELEMENTS AND THEOPERATOR HAVE BEEN CHANGEDMORE GENERALLY WE DESCRIBE AN ISOMORPHISM AS FOLLOWS  LET G1 ANDG2 BE TWO ALGEBRAIC OBJECTS EG GROUPS FIELDS VECTOR SPACESETC  LET  BE A BINARY OPERATION ON G1 AND LET CIRC BE THECORRESPONDING OPERATION ON G2  LET PHIMC  G1 RIGHTARROW G2BE A BF ONETOONE AND ONTO INVERTIBLE FUNCTION  FOR ANY XYIN G1 LET S  PHIX QQUAD TEXTANDQQUAD T  PHIYWHERE S IN G2 AND T IN G2  THEN PHI IS AN ISOMORPHISM IF PHIX  Y  PHIX CIRC PHIYNOTE THAT THE OPERATION ON THE LEFT TAKES PLACE IN G1 WHILE THEOPERATION ON THE RIGHT TAKES PLACE IN G2  ENDTEXTBOXBEGINEXAMPLE LABELEXMISO  USING THE VECTOR SPACE OF EXAMPLE REFEXMVS1 WE FIND  V OPLUS W 000000100000000010100010UNDER THE MAPPING PHIVBFWBF  VBF WBF WE FIND PHIV OPLUS W 000100010110WHICH IS THE SAME AS FOUND IN VW IN EXAMPLE REFEXMVS1  VECTORSPACE OPERATIONS ADDITION MULTIPLICATION BY A SCALAR ETC ON VOPLUS W HAVE EXACTLY ANALOGOUS RESULTS ON PHIVOPLUS W SO VOPLUS W AND VW ARE ISOMORPHICENDEXAMPLETHE DIRECT SUM V OPLUS W IS COMMONLY EMPLOYED BETWEEN ORTHOGONALVECTOR SPACES IN THE ISOMORPHIC FORM THAT IS AS THE SUM OF THEELEMENTS  THIS IS JUSTIFIED BECAUSE ORTHOGONAL SPACES ARE DISJOINTSEE EXERCISE REFEXORTHODISTHE FOLLOWING THEOREM INDICATES WHEN VW AND V OPLUS W AREISOMORPHICBEGINTHEOREM LABELTHMVWISO CITEPAGE 199NAYLORSELL LET V  AND W BE LINEAR   SUBSPACES OF A LINEAR SPACE S  THEN VW AND V OPLUS W ARE  ISOMORPHIC IF AND ONLY IF V AND W ARE DISJOINTENDTHEOREMBECAUSE OF THIS THEOREM WHEN V AND W ARE DISJOINT IT IS FREQUENTTO WRITE VW IN PLACE OF V OPLUS W AND VICE VERSA  CARE SHOULDBE TAKEN HOWEVER TO UNDERSTAND WHAT SPACE IS ACTUALLY INTENDEDBEGINEXERCISESITEM SHOW THAT THE SET VOPLUS W HAS THE SAME ALGEBRAIC STRUCTURE  AS DOES THE SET V  W ESTABLISHING THAT THE ISOMORPHISM HOLDITEM CITEP 200NAYLORSELL LET X  L2PIPI AND LET S1  LSPAN1COS TCOS 2TLDOTSQQUAD QQUAD S2 LSPANSIN T SIN 2T LDOTSITEM SHOW THAT S1 OPLUS S2 AND S1  S2 ARE ISOMORPHICITEM SHOW THAT DIMENSIONS1 OPLUS S2  DIMENSIONS1  DIMENSIONS2ITEM LET S BE A LINEAR SPACE AND ASSUME THAT S  S1  S2   CDOTS  SN WHERE THE SI ARE ARE MUTUALLY DISJOINT LINEAR  SUBSPACES OF S  LET BI BE A HAMEL BASIS OF SI  SHOW THAT  B  B1 CUP B2 CDOTS CUP BN IS A HAMEL BASIS FOR SITEM LABELEXORTHODIS SHOW THAT  BEGINENUMERATE  ITEM IF V AND W ARE ORTHOGONAL SUBSPACES THEN THEY ARE    DISJOINT  ITEM IF V AND W ARE DISJOINT THEY ARE NOT NECESSARILY    ORTHOGONAL  ENDENUMERATEITEM PROVE LEMMA REFLEMVWUNIQUEENDEXERCISESSECTIONPROJECTIONS AND ORTHOGONAL PROJECTIONSLABELSECPROJECTIONSAS POINTED OUT IN LEMMA REFLEMVWUNIQUE IF V AND W AREDISJOINT SUBSPACES OF A LINEAR SPACE S THEN ANY VECTOR XBF IN SCAN BE UNIQUELY WRITTEN AS  XBF   VBF  WBFWHERE VBF IN V AND WBF IN W  THIS REPRESENTATION ISILLUSTRATED IN FIGURE REFFIGDISJOINT2BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRDISJOINT2    CAPTIONDECOMPOSITION OF XBF INTO DISJOINT COMPONENTS    LABELFIGDISJOINT2  ENDCENTERENDFIGURELET US INTRODUCE PROJECTION INDEXPROJECTION OPERATOR PMC SRIGHTARROW V WITH THE FOLLOWING OPERATION FOR ANY XBF IN S WITHTHE DECOMPOSITION XBF  VBF  WBFLET P XBF  VBFTHAT IS THE PROJECTION OPERATOR RETURNS THAT COMPONENT OF XBFWHICH LIES IN V  IF XBFIN V TO BEGIN WITH THEN OPERATION BYP DOES NOT CHANGE THE VALUE OF XBF  THUS SINCE PXBF IN VWE SEE THAT PPXBF  PXBF  THIS MOTIVATES THE FOLLOWINGDEFINITIONBEGINDEFINITION  A LINEAR TRANSFORMATION P OF A LINEAR SPACE INTO ITSELF IS A BF    PROJECTION IF P2  P INDEXPROJECTIONENDDEFINITIONNOINDENT AN OPERATOR P SUCH THAT P2  P IS SAID TO BE EM  IDEMPOTENT INDEXIDEMPOTENTIF V IS A LINEAR SUBSPACE AND P IS AN OPERATOR THAT PROJECTS ONTOV THE PROJECTION OF A VECTOR XBF ONTO V IS SOMETIMES DENOTED ASXPROJ VTHE RANGE AND NULLSPACE OF A PROJECTION OPERATOR PROVIDE ADISJOINT DECOMPOSITION OF A VECTOR SPACE AS THE FOLLOWING THEOREMSHOWSBEGINTHEOREM LABELTHMRNP  LET P BE A PROJECTION OPERATOR DEFINED ON A LINEAR SPACE S  THEN THE RANGE AND NULLSPACE OF P ARE DISJOINT LINEAR SUBSPACES OF  S AND S  RANGEP  NULLSPACEP  THAT IS RANGEP AND  NULLSPACEP ARE ALGEBRAIC COMPLEMENTS  INDEXALGEBRAIC COMPLEMENTENDTHEOREMBEGINEXAMPLE  LET XT BE A SIGNAL WITH FOURIER TRANSFORM XOMEGA  THEN THE  TRANSFORMATION POMEGA0 OMEGA0 GEQ 0 DEFINED BY PXOMEGA  BEGINCASES  XOMEGA  TEXTFOR  OMEGA0 LEQ OMEGA LEQ OMEGA0 0  TEXTOTHERWISEENDCASESWHICH CORRESPONDS TO FILTERING THE SIGNAL WITH A BRICKWALLLOWPASS FILTER IS A PROJECTION OPERATIONENDEXAMPLEBEGINEXAMPLE  LET PT T GEQ 0 BE THE TRANSFORMATION ON THE FUNCTION XT DEFINED BY PTXT  BEGINCASES  XT  TEXTFOR T LEQ T LEQ T 0  TEXTOTHERWISEENDCASESTHIS IS A TIMETRUNCATION OPERATION AND IS A PROJECTIONENDEXAMPLEBEGINEXAMPLE  A MATRIX A IS SAID TO BE A EM SMOOTHING MATRIX IF THERE IS A  SPACE OF SMOOTH VECTORS V SUCH THAT FOR A VECTOR XBF IN V AXBF  XBFTHAT IS A SMOOTH VECTOR UNAFFECTED BY A SMOOTHING OPERATION  ALSOTHE LIMIT AINFTY  LIMPRIGHTARROW INFTY APEXISTS  AS AN ARBITRARY VECTOR THAT IS NOT ALREADY SMOOTH ISREPEATEDLY SMOOTHED IT BECOMES INCREASINGLY SMOOTH  BY THEREQUIREMENT THAT AXBF  XBF FOR XBF IN V IT IS CLEAR THAT THESET OF SMOOTH VECTORS IS IN FACT RANGEA AND A IS A PROJECTIONMATRIX  SMOOTHING MATRICES ARE DISCUSSED FURTHER INCITEGREVILLE1957GREVILLE1966ENDEXAMPLELET P BE A PROJECTION ONTO A CLOSED SUBSPACE V OF STHEN IP IS ALSO A PROJECTION SEE EXERCISEREFEXIMP   THEN WE CAN WRITE XBF  PXBF  IPXBFTHIS DECOMPOSES XBF INTO THE TWO PARTS P XBF IN VAND IPXBF IN WAS FIGURE REFFIGDISJOINT2 SUGGESTS THE SUBSPACES V AND WINVOLVED IN THE PROJECTION ARE NOT NECESSARILY ORTHOGONAL  HOWEVERIN MOST APPLICATIONS ORTHOGONAL SUBSPACES ARE NEEDED  THIS LEADS TOTHE FOLLOWING DEFINITIONBEGINDEFINITION  LET P BE A PROJECTION OPERATOR ON AN INNER PRODUCT SPACE S  P  IS SAID TO BE AN EM ORTHOGONAL PROJECTION IF ITS RANGE AND  INDEXORTHOGONAL PROJECTION  NULLSPACE ARE ORTHOGONAL RANGEP PERP NULLSPACEPENDDEFINITIONTHE NEED FOR AN ORTHOGONAL PROJECTION MATRIX IS PROVIDED BY THEFOLLOWING PROBLEM GIVEN A POINT XBF IN A VECTOR SPACE S AND ASUBSPACE V SUBSET S WHAT IS THE NEAREST POINT IN V TO XBFCONSIDER THE VARIOUS REPRESENTATIONS OF XBF SHOWN IN FIGUREREFFIGORTHOGPROJ1  AS SUGGESTED BY THE FIGURE DECOMPOSITION OFXBF AS XBF  VBF0  WBF0PROVIDES THE POINT VBF0IN V THAT IS CLOSEST TO XBF  THEVECTOR WBF0 IS ORTHOGONAL TO V WITH RESPECT TO THE INNERPRODUCT APPROPRIATE TO THE PROBLEM  OF THE VARIOUS WBF VECTORSTHAT MIGHT BE USED IN THE REPRESENTATION THE VECTORS WBF0WBF1 OR WBF2 IN THE FIGURE THE VECTOR WBF0 IS THE VECTOROF THE SHORTEST LENGTH AS DETERMINED BY THE NORM INDUCED BY THE INNERPRODUCT   PROOF OF THIS GEOMETRICALLY APPEALING AND INTUITIVE NOTIONIS PRESENTED IN THE NEXT SECTION AS THE PROJECTION THEOREM  BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRORTHOGPROJ1    CAPTIONORTHOGONAL PROJECTION FINDS THE CLOSEST POINT IN V TO      XBF    LABELFIGORTHOGPROJ1  ENDCENTERENDFIGUREIT IS DIFFICULT TO OVERSTATE THE IMPORTANCE OF THE NOTION OFPROJECTION  PROJECTION IS THE KEY CONCEPT OF MOST STOCHASTICFILTERING AND PREDICTION THEORY IN SIGNAL PROCESSING  CHAPTERREFCHAPVECTAP CONTAINS SEVERAL APPLICATIONS OF THIS IMPORTANTCONCEPTANOTHER VIEWPOINT OF THE PROJECTION THEOREM IS REPRESENTED IN FIGUREREFFIGORTHOGPROJ2  SUPPOSE THAT V IS THE SPAN OF THE BASISVECTORS PBF1PBF2 AS SHOWN  THEN THE NEAREST POINT TOXBF IN V IS THE POINT VBF0 AND THE VECTOR WBF0 IS THEDIFFERENCE  IF WBF0 IS ORTHOGONAL TO VBF0 THEN IT MUST BEORTHOGONAL TO PBF1 AND PBF2  BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRORTHOGPROJ2    CAPTIONORTHOGONAL PROJECTION ONTO THE SPACE SPANNED BY SEVERAL VECTORS    LABELFIGORTHOGPROJ2  ENDCENTERENDFIGUREIF WE REGARD VBF0 AS AN APPROXIMATION TO XBF THAT MUST LIE INTHE SPAN OF PBF1 AND PBF2 THEN WBF0  XBF  VBF0IS THE APPROXIMATION ERROR  CONSIDER THE VECTORS PBF1 ANDPBF2 AS THE DATA FROM WHICH THE APPROXIMATION IS TO BE FORMEDTHEN EM THE LENGTH OF THE APPROXIMATION ERROR VECTOR WBF0 IS  MINIMIZED WHEN THE ERROR IS ORTHOGONAL TO THE DATA SUBSECTIONPROJECTION MATRICESLABELSECPROJMATINDEXPROJECTION MATRIXLET US RESTRICT OUR ATTENTION FOR THE MOMENT TO FINITEDIMENSIONALVECTOR SPACES  LET A BE AN MATSIZEMN MATRIX WRITTEN AS A  PBF1PBF2LDOTSPBFNAND LET THE SUBSPACE V BE THE COLUMN SPACE OF A  V  LSPANPBF1 PBF2 LDOTS PBFN  RANGEAASSUME THAT WE ARE USING THE USUAL INNER PRODUCT LA XBF YBFRA XBFH YBF  THEN AS WE SEE IN THE NEXT CHAPTER THEPROJECTION MATRIX PA THAT PROJECTS ORTHOGONALLY ONTO THE COLUMNSPACE OF A IS BEGINEQUATION PA  AAHA1AHLABELEQPROJMAT1ENDEQUATIONINDEXPROJECTION MATRIXBEGINTHEOREM LABELTHMSYMPROJ  ANY HERMITIAN SYMMETRIC MATRIX WITH P2  P IS AN ORTHOGONAL  PROJECTION MATRIXENDTHEOREMLOOKING AHEAD TO WHERE THESE CONCEPTS ARE DEFINED IT CAN BE SHOWNTHAT ANY SELFADJOINT BOUNDED LINEAR OPERATOR P WITH P2P IS APROJECTION OPERATORBEGINPROOF  THE OPERATION PXBF IS A LINEAR COMBINATION OF THE COLUMNS OF P  TO SHOW THAT P IS AN ORTHOGONAL PROJECTION WE MUST SHOW THAT XBF   PXBF IS ORTHOGONAL TO THE COLUMN SPACE OF P  FOR ANY VECTOR  PCBF IN THE COLUMN SPACE OF P XBF PXBFH PCBF  XBFHPP2CBF  0SO XBF  PXBF IS ORTHOGONAL TO THE COLUMN SPACE OF PENDPROOFIT WILL OCCASIONALLY BE USEFUL TO DO THE PROJECTION USING A WEIGHTEDINNER PRODUCT  LET THE INNER PRODUCT BEBEGINEQUATION LA XBF YBFRAW  XBFH W YBFLABELEQWIPPRENDEQUATIONWHERE W IS A POSITIVEDEFINITE HERMITIAN MATRIX  THE INDUCED NORMIS  XBFW2  LA XBF XBFRAW  XBFH W XBFLET A BE AN MATSIZEMN MATRIX AS BEFORE  THEN THE PROJECTIONMATRIX WHICH PROJECTS ORTHOGONALLY ONTO THE COLUMN SPACE OF A WHERETHE ORTHOGONALITY IS ESTABLISHED USING THE INNER PRODUCTREFEQWIPPR IS THE MATRIXBEGINEQUATION PAW  AAH W A1 AH WLABELEQPROJMAT2ENDEQUATIONBEGINEXERCISESITEM IF VBF IS A VECTOR SHOW THAT THE MATRIX WHICH PROJECTS ONTO LSPANVBF IS PV  FRACVBF VBFHVBFHVBFITEM SHOW THAT THE MATRIX PA IN REFEQPROJMAT1 IS A PROJECTION  MATRIXITEM FOR THE PROJECTION MATRIX PAW INREFEQPROJMAT2  BEGINENUMERATE  ITEM SHOW THAT PAW2  PAW  ITEM SHOW THAT PAWPERP  IPAW IS ORTHOGONAL TO    PAW USING THE WEIGHTED INNER PRODUCT  THAT IS    PAWH W PAWPERP  0  ENDENUMERATE  ITEM LET  PBF1  BEGINBMATRIX 123  4 ENDBMATRIX QQUADPBF2  BEGINBMATRIX 4  2  6  7 ENDBMATRIX QQUADPBF3  BEGINBMATRIX 3  4  2  1 ENDBMATRIXAND  XBF  BEGINBMATRIX 1  2  3  7 ENDBMATRIXDETERMINE THE NEAREST VECTOR XBFHAT INLSPANPBF1PBF2PBF3  ALSO DETERMINE THE ORTHOGONALCOMPLEMENT OF XBF IN LSPANPBF1PBF2PBF3ITEM LET A BE A MATRIX WHICH CAN BE FACTORED ASBEGINEQUATIONA  U SIGMA VHLABELEQSVDEXER1ENDEQUATIONSUCH THAT UAH UA  I QQUAD VAH VA  I AND SIGMAA IS  DIAGONAL MATRIX WITH REAL VALUESTHE FACTORIZATION IN REFEQSVDEXER1 IS THE SINGULAR VALUEDECOMPOSITION SEE CHAPTER REFCHASVDSHOW THAT PA  PUA  ITEM TWO ORTHOGONAL PROJECTION OPERATORS PA AND PB ARE SAID TO  BE ORTHOGONAL IF PAPB  0  SHOW THAT  BEGINENUMERATE  ITEM PA AND PB ARE ORTHOGONAL IF AND ONLY IF THEIR RANGES ARE    ORTHOGONAL  ITEM PAPB IS A PROJECTION OPERATOR IF AND ONLY IF PA AND    PB  ARE ORTHOGONAL  ENDENUMERATEITEM SHOW THAT THE CBF WHICH MINIMIZES REFEQWPROJ1 IS CBF  AHWA1AHW XBFSO THAT THE WEIGHTED PROJECTION OF XBF IS ACBF AAHWA1AHW XBFITEM PROVE THEOREM REFTHMRNPITEM LABELEXIMP IF P IS A PROJECTION OPERATOR SHOW THAT IP  IS A PROJECTION OPERATOR  DETERMINE THE RANGE AND NULLSPACE OF  IPITEM LET S BE A VECTOR SPACE AND LET V1V2LDOTS VN BE  LINEAR SUBSPACES SUCH THAT VI IS DISJOINT FROM SUMJ NEQ I  VJ FOR EACH I  LET PJ BE THE PROJECTION ON S FRO WHICH  RANGEPJ  VJ AND NULLSPACEPJ  SUMJ NEQ K VK  DEFINE AN OPERATOR T  LAMBDA1 P1  LAMBDA2 P2  CDOTS  LAMBDAN PNBEGINENUMERATEITEM SHOW THAT IF XBF IN VJ THEN T XBF  LAMBDAJ XBFITEM SHOW THAT T IS A PROJECTION IF AND ONLY LAMBDAJ IS EITHER  0 OR 1ENDENUMERATELET P1P2LDOTS PM BE A SET OF ORTHOGONAL PROJECTIONS WITH PIPJ  0 FOR I NEQ JBEGINENUMERATEITEM SHOW THAT Q  P1  P2  CDOTS  PM IS AN ORTHOGONAL  PROJECTIONITEM WHAT HAPPENS IF P1 P2 NEQ 0ENDENUMERATEITEM LET A AND B BE MATRICES SUCH THAT AHB  0 SEEREFEQABORTHOG  THEN V  RANGEA AND W  RANGEB AREORTHOGONAL  BEGINENUMERATEITEM SHOW THAT PA  I  PBITEM SHOW THAT A XBF CAN BE DECOMPOSED AS XBF  PA XBF  PBXBF  PA XBF  IPAXBFENDENUMERATEENDEXERCISESSECTIONTHE PROJECTION THEOREMBEGINQUOTESOURCEMICHAEL SPIVAKEM CALCULUS ON MANIFOLDS  IMPORTANT ATTRIBUTES OF MANY FULLY EVOLVED MAJOR THEOREMS  BEGINENUMERATE  ITEM IT IS TRIVIAL  ITEM IT IS TRIVIAL BECAUSE THE TERMS APPEARING IN IT HAVE BEEN    PROPERLY DEFINED  ITEM IT HAS SIGNIFICANT CONSEQUENCES  ENDENUMERATEENDQUOTESOURCETHE MAIN PURPOSE OF THIS SECTION IS TO PROVE THE GEOMETRICALLYINTUITIVE NOTION INTRODUCED IN THE PREVIOUS SECTION THE POINT VBF0IN V THAT IS CLOSEST TO A POINT XBF IS THE ORTHOGONAL PROJECTIONOF XBF ONTO VBEGINTHEOREM LABELTHMPROJ THE PROJECTION THEOREM  INDEXPROJECTION THEOREM CITELUENBERGER1969 LET S BE A  HILBERT SPACE AND LET V BE A CLOSED SUBSPACE OF S  FOR ANY  VECTOR XBF IN S THERE EXISTS A EM UNIQUE VECTOR VBF0 IN  V CLOSEST TO XBF THAT IS  XBF VBF0 LEQ  XBF  VBF   FOR ALL VBF IN V  FURTHERMORE THE POINT VBF0 IS THE  MINIMIZER OF  XBF VBF0 IF AND ONLY IF XBF  VBF0 IS  ORTHOGONAL TO VENDTHEOREMTHE IDEA BEHIND THE THEOREM IS SHOWN IN FIGURE REFFIGPROJ1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPROJ1    CAPTIONTHE PROJECTION THEOREM    LABELFIGPROJ1  ENDCENTERENDFIGUREBEGINPROOF  THERE ARE SEVERAL ASPECTS OF THIS THEOREM    BEGINENUMERATE  ITEM THE FIRST AND MOST TECHNICAL ASPECT IS THE EM EXISTENCE    OF THE MINIMIZING POINT VBF0  ASSUME XBF NOT IN V AND    LET DELTA  INFVBF IN V XBF  VBF  WE NEED TO SHOW    THAT THERE IS A VBF0 IN V WITH XBF  VBF0 DELTA    LET VBFI BE A SEQUENCE OF VECTORS IN V SUCH THAT X     VI RIGHTARROW DELTA  WE WILL SHOW THAT VI IS A    CAUCHY SEQUENCE HENCE HAS A LIMIT IN S  BY    REFEQPARALLELOGRAM VBFJXBF  XBF  VBFI 2  VBFJ XBF XBFVBFI2  2VBFJ  XBF2  2XBFVBFI2THE LATTER CAN BE REARRANGED AS VBFJ  VBFI2  2VBFJ  XBF2  2XBFVBFI2 4XBF  VBFIVBFJ22SINCE S IS A VECTOR SPACE VBFIVBFJ2 IN S  ALSO BY THEDEFINITION OF DELTA XBF VBFIVBFJ2 GEQ DELTASO THAT  VBFI  VBFJ2 LEQ 2VBFJ  XBF  2XBFVBFI 4DELTA2 THEN SINCE VBFI IS DEFINED SO THAT VBFJXBF RIGHTARROWDELTA2 WE CONCLUDE THAT VBFIVBFJ2 RIGHTARROW 0SO VBFI IS A CAUCHY SEQUENCE  SINCE V IS A HILBERT SPACE ASUBSPACE OF S THE LIMIT EXISTS AND V0 IN VITEM LET US NOW SHOW THAT IF VBF0 MINIMIZES  XBF  VBF0  THEN XBF    VBF0 PERP V  LET VBF0 BE THE NEAREST  VECTOR TO XBF IN V LET VBF BE  A UNITNORM VECTOR IN V SUCH THAT CONTRARY TO THE STATEMENT OF  THE THEOREM LA XBF  VBF0 VBFRA  DELTA NEQ 0LET ZBF  VBF0  DELTA VBF IN V FOR SOME NUMBER DELTATHEN BEGINALIGNED XBF  ZBF 2   XBF  VBF0 2  2 REALLA XBF VBF0 DELTA VBF RA   DELTA VBF2    XBF  VBF02  DELTA2   XBF  VBF02ENDALIGNEDTHIS IS A CONTRADICTION HENCE DELTA  0ITEM CONVERSELY SUPPOSE THAT XBFVBF0 PERP V  THEN FOR ANY VBFIN V WITH VBF NEQ VBF0BEGINALIGN XBF  VBF2   XBF  VBF0  VBF0  VBF2 NONUMBER   XBF  VBF02   VBF0  VBF2 LABELEQHOTH GEQ  XBF  VBF02ENDALIGNWHERE ORTHOGONALITY IS USED TO OBTAIN REFEQHOTHITEM INDEXUNIQUENESS UNIQUENESS OF THE NEAREST POINT IN V TO  XBF MAY BE SHOWN AS FOLLOWS  SUPPOSE THAT XBF  VBF1   WBF1  VBF2  WBF2 WHERE WBF1  XBF  VBF1 PERP V AND  WBF2  XBF  VBF2 PERP V FOR SOME VBF1 VBF2 IN V  THEN 0  VBF1  VBF2  WBF1  WBF2 OR VBF2  VBF1  WBF1WBF2BUT SINCE VBF2VBF1 IN V IT FOLLOWS THAT WBF1WBF2 INV SO WBF1  W2 HENCE VBF1 VBF2  ENDENUMERATEENDPROOFBASED ON THE PROJECTION THEOREM EVERY VECTOR IN A HILBERT SPACE SCAN BE EXPRESSED UNIQUELY AS THAT PART WHICH LIES IN A SUBSPACE VAND THAT PART WHICH IS ORTHOGONAL TO VBEGINTHEOREM CITELUENBERGER1969 LABELTHMHDECOMP  LET V BE A CLOSED LINEAR SUBSPACE OF A HILBERT SPACE S  THEN S  V OPLUS VPERPAND V  VPERPPERPENDTHEOREMTHE ISOMORPHIC INTERPRETATION OF THE DIRECT SUM IS IMPLIED IN THISNOTATIONBEGINPROOF  LET XBF IN S  THEN BY THE PROJECTION THEOREM THERE IS A UNIQUE  VBF0 IN V SUCH THAT  XBF  VBF0 LEQ XBF  VBF   FOR ALL VBF IN V AND WBF0  XBF VBF0 IN VPERP  WE CAN THUS  DECOMPOSE ANY VECTOR IN S INTO XBF  VBF0  WBF0 QQUADTEXTWITH  VBF0 IN V WBF0 INVPERPTO SHOW THAT V  VPERPPERP WE NEED TO SHOW ONLY THATVPERPPERP SUBSET V SINCE WE ALREADY KNOW BY THEOREMREFTHMORTHOGCOMP THAT V SUBSET VPERPPERP  LET XBF INVPERPPERP  WE WILL SHOW THAT IT IS ALSO TRUE THAT XBF IN VBY THE FIRST PART WE CAN WRITE XBF  VBF  WBF WHERE VBF INV AND WBF IN VPERP  BUT SINCE V SUBSET VPERPPERP WEHAVE VBF IN VPERPPERP SO THAT WBF  XBF  VBF IN VPERPPERPSINCE WBF IN VPERP AND WBF IN VPERPPERP WE MUST HAVEWBF PERP W OR W  ZEROBF THUS XBF  VBF IN VENDPROOFTHIS THEOREM APPLIES TO HILBERT SPACES WHERE BOTH COMPLETENESS AND ANINNER PRODUCT DEFINING ORTHOGONALITY ARE AVAILABLEBEGINEXERCISES ITEM PROVE LEMMA REFLEMISO1ITEM SHOW THAT IF V AND W ARE CLOSED SUBSPACES OF A HILBERT  SPACE S AND IF V PERP W THEN VW IS A CLOSED SUBSPACE OF  S NS P 297ENDEXERCISESSECTIONORTHOGONALIZATION OF VECTORSLABELSECGRAMSCHMITINDEXGRAMSCHMIDT PROCESS IN MANY APPLICATIONS COMPUTATIONSINVOLVING BASIS VECTORS ARE EASIER IF THE VECTORS ARE ORTHOGONAL  ITIS THEREFORE USEFUL TO BE ABLE TO TAKE A SET OF VECTORS T ANDPRODUCE AN ORTHOGONAL SET OF VECTORS T WITH THE SAME SPAN AS TTHIS IS WHAT THE GRAMSCHMIDT ORTHOGONALIZATION PROCEDURE DOES  THEGRAMSCHMIDT PROCEDURE CAN ALSO BE USED TO DETERMINE THE DIMENSION OFTHE SPACE SPANNED BY A SET OF VECTORS SINCE A VECTOR LINEARLYDEPENDENT ON OTHER VECTORS EXAMINED PRIOR IN THE PROCEDURE YIELDS AZERO VECTORGIVEN A SET OF VECTORS T   PBF1PBF2LDOTSPBFN WE WANT TO FINDA SET OF VECTORS T  QBF1QBF2ALLOWBREAKLDOTSALLOWBREAK QBFN  WITH N LEQ N SO THATLSPANQBF1QBF2LDOTSQBFN   LSPANPBF1PBF2LDOTSPBFN AND  LA QBFIQBFJ RA  DELTAIJASSUME THAT NONE OF THE PBFI VECTORS ARE ZERO VECTORSTHE PROCESS WILL BE DEVELOPED STEPWISE  THE NORM  CDOT  INTHIS SECTION IS THE INDUCED NORMBEGINENUMERATEITEM NORMALIZE THE FIRST VECTOR QBF1  FRACPBF1PBF1ITEM COMPUTE THE DIFFERENCE BETWEEN THE PROJECTION OF PBF2 ONTO  QBF1 AND PBF2  BY THE ORTHOGONALITY THEOREM THIS IS  ORTHOGONAL TO P1 EBF2  PBF2  FRACLA PBF2QBF1 RA  QBF1 2 QBF1   PBF2  LA PBF2QBF1 RA QBF1IF EBF2  0 THEN QBF2 IN LSPANQBF1 AND CAN BEDISCARDED WE WILL ASSUME THAT SUCH DISCARDS ARE DONE AS NECESSARY INWHAT FOLLOWS  IF EBF2 NEQ 0 THEN NORMALIZE QBF2  FRACEBF2EBF2THESE STEPS ARE SHOWN IN FIGURE REFFIGGS1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRGRAMSCHMIDT1    CAPTIONTHE FIRST STEPS OF THE GRAMSCHMIDT PROCESS    LABELFIGGS1  ENDCENTERENDFIGUREITEM AT THE NEXT STAGE A VECTOR ORTHOGONAL TO QBF1 AND QBF2  IS OBTAINED FROM THE ERROR BETWEEN PBF3 AND ITS PROJECTION ONTO  LSPANQBF1QBF2 EBF3  PBF3  LA PBF3QBF1 QBF1  LA PBF3 QBF2RA  QBF2THIS IS NORMALIZED TO PRODUCE QBF3 QBF3  FRACQBF3QBF3SEE FIGURE REFFIGGS2BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRGRAMSCHMIDT2    CAPTIONTHIRD STEP OF THE GRAMSCHMIDT PROCESS    LABELFIGGS2  ENDCENTERENDFIGUREITEM NOW PROCEED INDUCTIVELY  TO FORM THE NEXT ORTHOGONAL VECTOR  USING PBFK DETERMINE THE COMPONENT ORTHOGONAL TO ALL PREVIOUSLY  FOUND VECTORSBEGINEQUATION EBFK  PBFK  SUMI1K1 LA PBFKQBFIRA  QBFILABELEQGSFORMENDEQUATIONAND NORMALIZEBEGINEQUATION QBFK  FRACEBFKEBFK2LABELEQGSFORM2ENDEQUATIONENDENUMERATEBEGINEXAMPLE LABELEXMLEGENDREPOLY  THE SET OF FUNCTIONS 1TT2LDOTSTM DEFINED OVER 11  FORMS A LINEARLY INDEPENDENT SET LET THE INNER PRODUCT BE LA FGRA  INT11 FTGTDTBY THE GRAMSCHMIDT PROCEDURE WE FINDBEGINALIGNED Q0T  FRAC1SQRT2  Q1T  SQRT32 T  Q2T  FRAC3SQRT522T213  Q3T  FRAC5SQRT722T3  3T5  VDOTSENDALIGNEDTHE FUNCTIONS SO OBTAINED ARE KNOWN AS THE EM LEGENDRE POLYNOMIALSTHEY ARE FREQUENTLY WRITTEN WITHOUT THENORMALIZATION ASBEGINALIGNED Y0T  1  Y1T  T  Y2T  T213  Y3T  T3  3T5  VDOTSENDALIGNEDINDEXLEGENDRE POLYNOMIALIF WE CHANGE THE INNER PRODUCT TO INCLUDE A WEIGHTING FUNCTION LA FGRA  INT11 FRAC1SQRT1T2 FTGTDTTHEN THE ORTHOGONAL POLYNOMIALS OBTAINED BY APPLYING THE GRAMSCHMIDTPROCESS TO THE POLYNOMIALS 1TLDOTSTN ARE THE CHEBYSHEVPOLYNOMIALS DESCRIBED IN EXAMPLE REFEXMCHEBYPOL INDEXCHEBYSHEV POLYNOMIAL INDEXORTHOGONAL POLYNOMIALENDEXAMPLESUBSUBSECTIONA MATRIXBASED IMPLEMENTATIONFOR FINITEDIMENSIONAL VECTORS THE GRAMSCHMIDT PROCESS CAN BEREPRESENTED IN A MATRIX FORM  LET A  PBF1PBF2LDOTSPBFN BE A MATSIZEMN MATRIX  THE ORTHOGONALVECTORS OBTAINED BY THE GRAMSCHMIDT PROCESS ARE STACKED IN A MATRIXQ  QBF1QBF2LDOTSQBFN TO BE DETERMINED WHERE N N  WE LET THE UPPER TRIANGULAR MATRIX R HOLD THE INNER PRODUCTSAND NORMS FROM REFEQGSFORM AND REFEQGSFORM2 R  BEGINBMATRIX PBF1  LA PBF2QBF1RA  LA PBF3QBF1 RA  CDOTS LA PBFN QBF1 RA              EBF2           LA PBF3QBF2 RA  CDOTS            LA PBFNQBF2 RA                                     EBF3           CDOTS            LA PBFNQBF3 RA                                                                 VDOTS                                                          CDOTS  EBFN ENDBMATRIXTHE INNER PRODUCTS IN THE SUMMATION SUMI1K1 LAPBFKQBFIRA QBFI ARE REPRESENTED BY RSUBRANGE1K1K QMCSUBRANGE1K1HAMCK AND THE SUM IS THENQMCSUBRANGE1K1RSUBRANGE1K1K  WE THUS OBTAIN THEFACTORIZATION  INDEXMATRIX FACTORIZATIONSQR INDEXQR FACTORIZATIONUSING GRAMSCHMIDT INDEXFACTORIZATIONSSEEMATRIX FACTORIZATIONS A  QRALGORITHM REFALGQR1 ILLUSTRATES A SC MATLAB IMPLEMENTATION OFTHIS GRAMSCHMIDT PROCESSBEGINNEWPROGENVGRAMSCHMIDT    ALGORITHM QR    FACTORIZATIONGRAMSCHMIDT1MQR1GRAMSCHMIDT    ALGORITHMENDNEWPROGENVWITH THE OBSERVATION FROM REFEQGSFORM THAT PBFK  QBFK RKK  SUMI1K1 RIKQBFIWE NOTE THAT WE CAN WRITE A IN A FACTORED FORM AS A  QR AND THATQ SATISFIES QHQ  I  NOTE THAT BOXEDTEXTTHE MATRIX Q PROVIDES AN ORTHOGONAL BASIS TO THE COLUMN  SPACE OF A FOR FINITEDIMENSIONAL VECTORS THE COMPUTATIONS OF THE GRAMSCHMIDTPROCESS MAY BE NUMERICALLY UNSTABLE FOR POORLY CONDITIONED MATRICESEXERCISE REFEXMGS DISCUSSES A MODIFIED GRAMSCHMIDT WHILE OTHERMORE NUMERICALLY STABLE METHODS OF ORTHOGONALIZATION ARE EXPLORED INCHAPTER REFCHAPMATFACTBEGINEXERCISESITEM MODIFY ALGORITHM REFALGQR1 SO THAT IT ONLY RETAINS COLUMNS  OF Q THAT ARE NONZERO MAKING CORRESPONDING ADJUSTMENTS TO R  COMMENT ON THE PRODUCT QR IN THIS CASE GRAMSCHMIDT2MITEM MODIFY ALGORITHM REFALGQR1 TO COMPUTE A SET OF ORTHOGONAL  VECTORS WITH RESPECT TO THE WEIGHTED INNER PRODUCT LA XBF YBF  RA  XBFT W YBF FOR A POSITIVE DEFINITE SYMMETRIC MATRIX WITEM DETERMINE THE FIRST FOUR POLYNOMIALS ORTHOGONAL OVER 01  A  SYMBOLIC MANIPULATION PACKAGE IS RECOMMENDEDITEM USING A SYMBOLIC MANIPULATION PACKAGE WRITE A FUNCTION WHICH  PERFORMS THE GRAMSCHMIDT ORTHOGONALIZATION OF A SET OF FUNCTIONSITEM FOR THE FUNCTIONS SHOWN IN FIGURE REFFIGGSEX DETERMINE A  SET OF ORTHOGONAL FUNCTIONS SPANNING THE SAME SPACE USING THE  FUNCTIONS IN THE ORDER SHOWN   SOMETHING LIKE THE PROAKIS EXERCISE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRGRAMSCHMIDT3      CAPTIONFUNCTIONS TO ORTHOGONALIZE      LABELFIGGSEX    ENDCENTER  ENDFIGUREITEM MODIFIED GRAMSCHMIDT INDEXMODIFIED GRAMSCHMIDT THE  COMPUTATIONS OF THE GRAMSCHMIDT ALGORITHM CAN BE REORGANIZED TO BE  MORE STABLE NUMERICALLY  IN THESE MODIFIED COMPUTATIONS A COLUMN  OF Q AND A EM ROW OF R IS PRODUCED AT EACH ITERATION  THE  REGULAR GRAMSCHMIDT PROCESS PRODUCES A COLUMN OF Q AND A COLUMN  OF R AT EACH ITERATION  LET THE ROWS OF R BE DENOTED AS  RBFIT  BEGINENUMERATE  ITEM SHOW THAT FOR AN MATSIZEMN MATRIX A A  SUMI1K1  SUMIKN QBFI RBFIT  ZEROBFAKWHERE AK IS MATSIZEMNK1ITEM LET AK  EBFK B AND EXPLAIN WHY THE KTH COLUMN OF  Q AND THE KTH ROW OF R ARE GIVEN BY RKK   EBFK QQUAD QBFK  EBFKRKK QQUADRKK1LDOTSRKN  QBFKTBITEM THEN SHOW THAT THE NEXT ITERATION CAN BE STARTED BY COMPUTING A  SUMI1K QBFI RBFIT  ZEROBF AK1WHERE AK1  B  QBFKRKK1LDOTSRKKNITEM CODE THE MODIFIED GRAMSCHMIDT ALGORITHM IN SC MATLAB  ENDENUMERATEENDEXERCISESSECTIONSOME FINAL TECHNICALITIES FOR INFINITEDIMENSIONAL SPACESTHE CONCEPT OF BASIS THAT WAS INTRODUCED IN SECTIONREFSECHAMELBASIS WAS BASED UPON THE STIPULATION THAT LINEARCOMBINATIONS ARE EM FINITE SUMS  WITH THE ADDITIONAL CONCEPTS OFORTHOGONALITY AND NORMALITY WE CAN INTRODUCE A SLIGHTLY MODIFIEDNOTION OF A BASIS  A SET T  PBF1PBF2LDOTS IS SAID TO BEORTHONORMAL INDEXORTHONORMAL BASIS IF LA XBFI XBFJRA DELTAIJ  FOR AN ORTHONORMAL SET T IT CAN BE SHOWN THAT THEINFINITE SUM SUMI1INFTY CI PBFICONVERGES IF AND ONLY IF THE SERIES SUMI1N CI2 CONVERGESAN ORTHONORMAL SET OF BASIS FUNCTIONS PBF1PBF2LDOTS ISSAID TO BE A BF COMPLETEFOOTNOTEA COMPLETE SET OF FUNCTIONS IS  DIFFERENT FROM COMPLETE SPACE IN WHICH EVERY CAUCHY SEQUENCE HAS A  LIMIT  SET INDEXCOMPLETE SET FOR A HILBERT SPACE S IF EVERYXBF IN S CAN BE REPRESENTED AS XBF  SUMI1INFTY CI PBFIFOR SOME SET OF COEFFICIENTS CI  SEVERAL SETS OF COMPLETE BASISFUNCTIONS ARE PRESENTED IN CHAPTER REFCHAPVECTAP AFTER A MEANSHAS BEEN PRESENTED FOR FINDING THE COEFFICIENTS CI  A COMPLETESET OF FUNCTIONS WILL BE CALLED A BF BASIS MORE STRICTLY ANORTHONORMAL BASIS  THE BASIS AND THE HAMEL BASIS ARE NOT IDENTICALFOR INFINITEDIMENSIONAL SPACES  IN PRACTICE IT IS THE BASIS NOTTHE HAMEL BASIS WHICH IS OF MOST USE  IT CAN BE SHOWN THAT ANYORTHONORMAL BASIS IS A SUBSET OF A HAMEL BASISIN FINITE DIMENSIONS NONE OF THESE ISSUES HAVE ANY BEARING  ANORTHONORMAL HAMEL BASIS EM IS  AN ORTHONORMAL BASIS  ONLY THENOTION OF BASIS NEEDS TO BE RETAINED FOR FINITE DIMENSIONALSPACES  IN THE FUTURE WE WILL DROP THE ADJECTIVE HAMEL AND REFERONLY TO A BASIS FOR A FINITEDIMENSIONAL VECTOR SPACEANOTHER CONCEPT THAT WE HAVE DANCED AROUND UP TO THIS POINT BUT FORWHICH THE STUDENT SHOULD HAVE SUFFICIENT MATURITY BY NOW IS THE NOTIONOF A DENSE SETBEGINDEFINITION  LET XD BE A CLOSED METRIC SPACE AND LET D SUBSET X  THEN  D IS BF DENSE IN X IF FOR EACH X IN X AND EVERY EPSILON   0 THERE IS A POINT D IN D SUCH THAT  X  D  EPSILONENDDEFINITIONTHE POINT OF A DENSE SET D IS THAT EVERY ELEMENT IN THE LARGER SETX IS SUFFICIENTLY CLOSE FOR ANY MEASURE OF SUFFICIENCY TO ANELEMENT OF D  ANOTHER DEFINITION OF A DENSE SET IS A SET DSUBSET X IS DENSE IN X IF THE CLOSE OF D IS XTHE MOST FAMOUS EXAMPLE OF DENSE SETS IS THE SET OF RATIONAL NUMBERSAS A SUBSET OF THE REAL NUMBERS  EVERY REAL NUMBER IS ARBITRARILYCLOSE TO SOME RATIONAL NUMBER  THE POINT OF DENSE SETS IS THAT IN MANY CASES WE CAN FOCUS ATTENTIONON THE DENSE SUBSET D WHICH IS USUALLY MUCH SMALLER THAN THEORIGINAL SET X  STATEMENTS WHICH ARE TRUE ON D CAN OFTEN BEEXTENDED TO STATEMENTS WHICH ARE TRUE ON XWHILE ON THIS TOPIC WE ROUND OUT OUR DISCUSSION WITH THE FOLLOWINGDEFINITION  BEGINDEFINITION  A NORMED SPACE X IS BF SEPARABLE IF IT CONTAINS A COUNTABLE  DENSE SETENDDEFINITIONTHE SET RBB IS SEPARABLE THE RATIONAL NUMBERS ARE COUNTABLE  ASSOME OTHER EXAMPLES OF SEPARABLE AND NONSEPARABLE SETS WE PRESENT THEFOLLOWING FOR PROOFS SEE CITEPAGE 43LUENBERGER1968BEGINENUMERATEITEM LET LP SPACES WITH P  INFTY ARE SEPARABLE LINFTY IS  NOT SEPARABLEITEM THE LP SPACES WITH P  INFTY ARE SEPARABLE  LINFTY  IS NOT SEPARABLEITEM THE SPACE CAB IS SEPARABLEENDENUMERATESETEXSECTREFSECNORM1BEGINEXERCISESEXSKIPITEM WE WILL EXAMINE THE LINFTY METRIC TO GET A SENSE AS TO WHY  IT SELECTS THE MAXIMUM VALUE  GIVEN THE VECTOR XBF  12345  6 COMPUTE THE LP METRIC DPXBFZEROBF FOR  P12410100INFTY  COMMENT ON WHY DPXBFZEROBF  RIGHTARROW MAXXI AS PRIGHTARROW INFTYITEM LET X BE AN ARBITRARY SET  SHOW THAT THE FUNCTION DEFINED BY DXY  BEGINCASES  1  X  Y   0  X NEQ 0ENDCASESIS A METRICITEM VERIFY THAT THE HAMMING DISTANCE DHXBFYBF INTRODUCED IN  EXAMPLE REFEXMHD1 IS A METRICITEM VERIFY THAT THE CODE SPACE OF EXAMPLE REFEXMCODESPACE IS A  METRIC SPACEITEM PROOF OF THE TRIANGLE INEQUALITY  BEGINENUMERATE  ITEM FOR XY IN RBB PROVE THE TRIANGLE INEQUALITY IN THE FORM XY LEQ X  YWHAT IS THE CONDITION FOR EQUALITY  ITEM FOR XBF YBF IN RBBN PROVE THE TRIANGLE INEQUALITY  XBFYBF  LEQ  XBF    YBF WHERE  CDOT  IS THE USUAL EUCLIDEAN NORM  HINT USE THE FACT THATSUMI1N XI YI LEQ  XBF  YBF  IE THE CAUCHYSCHWARZ INEQUALITY INDEXCAUCHYSCHWARZ INEQUALITY  ENDENUMERATEITEM LET XD BE A METRIC SPACE  SHOW THAT DBXY  FRACDXY1DXYIS A METRIC ON X  WHAT SIGNIFICANT FEATURE DOES THIS METRICPOSSESSITEM LET XD BE A METRIC SPACE  SHOW THAT DMXY  MIN1DXYIS A METRIC ON X  WHAT SIGNIFICANT FEATURE DOES THIS METRICPOSSESSITEM IN DEFINING THE METRIC OF THE SEQUENCE SPACE  LINFTY0INFTY IN REFEQLINFSEQ SUP WAS USED INSTEAD  OF MAX  TO SEE THE NECESSITY OF THIS DEFINITION DEFINE THE  SEQUENCES XBF AND YBF BY XN  FRAC1N1 QQUAD YN  FRACNN1SHOW THAT DINFTYXY  XNYN FOR ALL N GEQ 1ITEM FOR THE METRIC SPACE RBBNDP SHOW THAT DPXBFYBF  IS DECREASING WITH P  THAT IS DPXBFYBF GEQ  DQXBFYBF IF P LEQ Q  HINT TAKE THE DERIVATIVE WITH  RESPECT TO P AND SHOW THAT IT IS LEQ 0  USE THE EM LOG SUM    INEQUALITY CITECOVERTM1991 INDEXINEQUALITIESLOG SUM  WHICH STATES INDEXLOG SUM INEQUALITY THAT FOR NONNEGATIVE  SEQUENCES A1 A2 LDOTS AN AND B1 B2 LDOTS BN SUMI1N AI LOG LEFTFRACAIBIRIGHT GEQLEFTSUMI1N AIRIGHT LOG FRACSUMI1N AISUMI1N BIUSE BI  1 AND AI  XI  YIP  ALSO USE THE FACT THAT FOR ANYNONNEGATIVE SEQUENCE ALPHAI SUCH THAT SUMI1N ALPHAI 1 THE MAXIMUM VALUE OF  SUMI1N ALPHAI LOG ALPHAIIS 0ITEM IF REQUIREMENT M3 IN THE DEFINITION OF A METRIC IS RELAXED TO  THE REQUIREMENT DXY 0 TEXT IF  XYALLOWING THE POSSIBILITY THAT DXY0 EVEN WHEN X NEQ Y THEN AEM PSEUDOMETRIC IS OBTAINED  INDEXPSEUDOMETRIC  LET FMC  X RIGHTARROW RBB BE AN ARBITRARY FUNCTION DEFINED ON ASET X  SHOW THAT DXY  FXFY IS A PSEUDOMETRICEXSKIPITEM SHOW THAT IF A AND B ARE OPEN SETS  BEGINENUMERATE  ITEM A CUP B IS OPEN  ITEM A CAP B IS OPEN  ENDENUMERATEITEM DEVISE AN EXAMPLE TO SHOW THAT THE UNION OF AN INFINITE NUMBER  OF CLOSED SETS NEED NOT BE CLOSEDITEM LET  B  TEXTALL POINTS  P INRBB2 TEXT WITH  0  P LEQ 2 CUP TEXTTHE POINT  04BEGINENUMERATEITEM DRAW THE SET BITEM DETERMINE THE BOUNDARY OF BITEM DETERMINE THE INTERIOR OF BENDENUMERATEITEM EXPLAIN WHY THE SET OF REAL NUMBERS IS BOTH OPEN AND CLOSEDITEM DETERMINE INF AND SUP FOR THE FOLLOWING SETS OF REAL NUMBERS A  04 QQUAD B  0INFTY QQUAD C  INFTY5ITEM SHOW THAT THE BOUNDARY OF A SET S IS A CLOSED SETITEM SHOW THAT THE BOUNDARY OF A SET  S IS THE INTERSECTION OF THE  CLOSURE OF S AND THE CLOSURE OF THE COMPLEMENT OF SITEM SHOW THAT S SUBSET RBBN IS CLOSED IF AND ONLY IF EVERY  CLUSTER POINT FOR S BELONGS TO S  EXSKIPITEM FIND LIMSUPNRIGHTARROW INFTY AN AND  LIMINFNRIGHTARROW INFTY AN FOR  BEGINENUMERATE  ITEM AN  COSFRAC2PI3 N  ITEM AN  COSSQRT2 N  ITEM AN  2 1N3  2N  ITEM AN  N21N  ENDENUMERATEITEM IF LIMSUPNRIGHTARROW INFTY AN  A AND  LIMSUPNRIGHTARROW INFTY BN  B THEN IS IT NECESSARILY TRUE  THAT LIMSUPNRIGHTARROW INFTY ANBN  ABITEM SHOW THAT IF XN IS A SEQUENCE SUCH THAT  DXN1XN  C RNFOR 0 LEQ R  1 THEN XN IS A CAUCHY SEQUENCE  BUCK P 53ITEM LET PBFN  XNYNZN IN RBB3  SHOW THAT IF  PBFN IS A CAUCHY SEQUENCE USING THE METRIC DPBFJ PBFK  SQRTXJ  XK2  YJ  YK2  ZJ   ZK2THEN SO ARE THE SEQUENCES  XN YN AND ZN USINGTHE METRIC DXJXK  XJ  XKITEM SHOW THAT IF A SEQUENCE XN IS CONVERGENT THEN IT IS A CAUCHY  SEQUENCE INDEXCAUCHY SEQUENCECONVERGENCEITEM SHOW THAT THE SEQUENCE XN  INT1N FRACCOS TT2  DT  IS CONVERGENT USING THE METRIC DXY  XY  HINT SHOW THAT  XN IS A CAUCHY SEQUENCE  USE THE FACT THAT INTFRACCOS TT2DT LEQ INTFRAC1T2DTNOTE THIS IS AN EXAMPLE OF KNOWING THAT A SEQUENCE CONVERGESWITHOUT KNOWING WHAT IT CONVERGES TO ITEM SHOW THAT IF XN IS A CAUCHY SEQUENCE THEN XN IS   CONVERGENT PROVIDED THAT THE LIMIT EXISTSITEM THE FACT THAT A SEQUENCE IS CAUCHY DEPENDS UPON THE METRIC  EMPLOYED  LET FNT BE THE SEQUENCE OF FUNCTIONS DEFINED IN  REFEQFNSEQ IN THE METRIC SPACE CABDINFTY WHERE DINFTYFG  SUPT FT  GTSHOW THAT DINFTYFNFM  FRAC12FRACN2M QQUAD MNHENCE CONCLUDE THAT IN THIS METRIC SPACE FN IS NOT A CAUCHYSEQUENCEITEM IN THIS PROBLEM WE WILL SHOW THAT THE SET OF CONTINUOUS  FUNCTIONS IS COMPLETE WITH RESPECT TO THE UNIFORM SUP NORM  LET   FNT BE A CAUCHY SEQUENCE OF CONTINUOUS FUNCTIONS  LET  FT BE THE POINTWISE LIMIT OF FNT  FOR ANY EPSILON  0 LET N BE CHOSEN SO THAT MAX FNTFMT LEQ  EPSILON3  SINCE FKT IS CONTINUOUS THERE IS A D0  SUCH  THAT FTDELTA  FT  EPSILON3 WHENEVER DELTA LEQ D    FROM THIS CONCLUDE THAT  FTDELTA  FT  EPSILONAND HENCE THAT FT IS CONTINUOUSITEM FIND THE ESSENTIAL SUPREMUM OF THE FUNCTION XT DEFINED BY XT  BEGINCASES  SINPI T  T IN 11 T NEQ 0   3  T0ENDCASESEXSKIPSETEXSECTREFSECVS1ITEM AN EQUIVALENT DEFINITION FOR LINEAR INDEPENDENCE FOLLOWS A SET  T IS LINEARLY INDEPENDENT IF FOR EACH VECTOR XBF IN T XBF  IS NOT A LINEAR COMBINATION OF THE POINTS IN THE SET T    XBF THAT IS THE SET T WITH THE VECTOR XBF REMOVED  SHOW  THAT THIS DEFINITION IS EQUIVALENT TO THAT OF DEFINITION  REFDEFLININD  ITEM LET S BE A FINITEDIMENSIONAL VECTOR SPACE WITH    DIMENSIONS  M  SHOW THAT EVERY SET CONTAINING M1 POINTS    IS LINEARLY DEPENDENT  HINT USE INDUCTION  ITEM SHOW THAT IF T IS A SUBSET OF A VECTOR SPACE S WITH    LSPANTS THEN T CONTAINS A HAMEL BASIS OF S  ITEM LET S DENOTE THE SET OF ALL SOLUTIONS OF THE FOLLOWING DIFFERENTIAL    EQUATION DEFINED ON C30INFTY  SEE DEFINITION    REFDEFCLASSCK  INDEXCKCLASS CK FRACD3 XDT3  B FRACD2XDT2  C FRACDXDT  DX     0SHOW THAT S IS A LINEAR SUBSPACE OF C30INFTYITEM LET S BE L202PI AND LET T BE THE SET OF ALL  FUNCTIONS XNT  EJNT FOR N01LDOTS  SHOW THAT T IS  LINEARLY INDEPENDENT  CONCLUDE THAT L202PI IS AN INFINITE  DIMENSIONAL SPACE  HINT ASSUME THAT C1 EJ N1 T  C2 EJ    N2 T  CDOTS  CM EJ NM T  0 FOR NI NEQ NJ WHEN I  NEQ J  DIFFERENTIATE M1 TIMES AND USE THE PROPERTIES OF  VANDERMONDE MATRICES SECTION REFSECVANDERMONDE  ITEM LABELEXLINIDPOLY SHOW THAT THE SET 1TT2LDOTSTM    IS A LINEARLY INDEPENDENT SET HINT THE FUNDAMENTAL THEOREM OF    INDEXFUNDAMENTAL THEOREM OF ALGEBRA    ALGEBRA STATES THAT A POLYNOMIAL FX OF DEGREE M HAS EXACTLY    M ROOTS COUNTING MULTIPLICITYKEENER P 3EXSKIPSETEXSECTREFSECNORMVSITEM LABELEXTINEQBK SHOW THAT IN A NORMED LINEAR SPACE BOXED X  Y LEQ XYITEM SHOW THAT A NORM IS A CONVEX FUNCTION  SEE SECTION  REFSECCONVFUNCITEM SHOW THAT EVERY CAUCHY SEQUENCE  XN IN A NORMED LINEAR  SPACE IS BOUNDED INDEXCAUCHY SEQUENCEBOUNDED INDEXBOUNDED    SEQUENCEITEM USING THE TRIANGLE INEQUALITY SHOW THAT ZX LEQ ZY  YXITEM LET X BE THE SPACE OF EM FINITELY NONZERO SEQUENCES  XBF  X1X2X3LDOTSXN00LDOTS  DEFINE THE NORM ON  X AS XBF  MAXIXI  LET XBFN BE A POINT IN X A  SEQUENCE DEFINED BY XBFN   1FRAC12FRAC13 LDOTS    FRAC1N100LDOTSBEGINENUMERATEITEM SHOW THAT THE SEQUENCE XBFN IS A CAUCHY SEQUENCEITEM SHOW THAT X IS NOT COMPLETE INDEXCOMPLETE METRIC SPACEENDENUMERATEITEM LET P BE IN THE RANGE 0  P  1 AND CONSIDER THE SPACE  LP01 OF ALL FUNCTIONS WITH   X  INT01 XTPDT  INFTYSHOW THAT X IS NOT A NORM ON LP01   HOWEVER SHOW THAT DXY   XY IS A METRIC  HINT FOR A REAL NUMBER ALPHASUCH THAT 0 LEQ ALPHA LEQ 1 NOTE THAT ALPHA LEQ ALPHAP LEQ1ITEM LET S BE A NORMED LINEAR SPACE  SHOW THAT THE NORM FUNCTION  CDOTMC  S RIGHTARROW RBB IS CONTINUOUS  HINT SEE  EXERCISE REFEXTINEQBK  ITEM FOR EACH OF THE INEQUALITY RELATIONSHIPS BETWEEN NORMS IN  REFEQNORMCOMP DETERMINE A VECTOR XBF IN RBBN FOR WHICH  EACH INEQUALITY IS ACHIEVED WITH EQUALITY  EXSKIPSETEXSECTREFSECINNERPROD1 KEENER P 8ITEM COMPUTE THE INNER PRODUCTS LA FG RA FOR THE FOLLOWINGUSING THE DEFINITION LA FG RA  INT01 FTGTDTBEGINENUMERATEITEM FT  T2  2T GT  T1ITEM FT  ET GT T1ITEM FT  COS2PI T GT  SIN2PI TENDENUMERATEITEM COMPUTE THE INNER PRODUCTS XBFT YBF OF THE FOLLOWING USING  THE EUCLIDEAN INNER PRODUCT  BEGINENUMERATE  ITEM XBF  1234T YBF  2341T  ITEM XBF  23 YBF  12T  ENDENUMERATEITEM DETERMINE WHICH OF THE FOLLOWING DETERMINES AN INNER PRODUCT  OVER THE SPACE OF REAL CONTINUOUS FUNCTIONS WITH CONTINUOUS FIRST  DERIVATIVES I  LA FGRA  INT01 FTGTDT  F0G0 QQUADQQUADII   LA FGRA  INT01 FTGTDTEXSKIPSETEXSECTREFSECINDNORM  ITEM SHOW THAT FOR AN INDUCED NORM CDOT OVER A REAL VECTOR SPACE    BEGINENUMERATE    ITEM THE EM PARALLELOGRAM LAW IS TRUE    BEGINEQUATION      LABELEQPARALLELOGRAM       XY 2   XY2  2X2 2Y2    ENDEQUATION    IN TWODIMENSIONAL GEOMETRY AS SHOWN IN FIGURE    REFFIGPARALLELOGRAM THE RESULT SAYS THAT THE SUM OF SQUARES    OF THE LENGTHS OF THE DIAGONALS IS EQUAL TO TWICE THE SUM OF THE    SQUARES OF THE ADJACENT SIDES A SORT OF TWOFOLD PYTHAGOREAN    THEOREMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPARALLEL    CAPTIONTHE PARALLELOGRAM LAW    LABELFIGPARALLELOGRAM  ENDCENTERENDFIGUREITEM SHOW THAT  LA XYRA  FRACX  Y2  X  Y24 THIS IS KNOWN AS THE EM POLARIZATION IDENTITYINDEXPOLARIZATION  IDENTITYENDENUMERATEEXSKIPSETEXSECTREFSECCSITEM FOR THE INNER PRODUCE LA FGRA  INT01 FTGTDT  VERIFY THE CAUCHYSCHWARZ INEQUALITY IF  BEGINENUMERATE  ITEM FT  ET GT T1  ITEM FT  ET GT  5 ET  ENDENUMERATEITEM SHOW THAT THE INEQUALITY REFEQANGLEBOUND IS TRUEEXSKIPSETEXSECTREFSECDIRVECITEM PROVE LEMMA REFLEMPYTH  ITEM LET X1T  3T2  1 X2T 5T3  3T AND X3T     2T2  T AND DEFINE THE INNER PRODUCT AS LA FGRA     INT11 FTGTDT  COMPUTE THE ANGLES OF EACH PAIRWISE    COMBINATION OF THESE FUNCTIONS AND IDENTIFY FUNCTIONS THAT ARE    ORTHOGONAL  ITEM LET  BEGINALIGNEDXBF1  1 2 4 2T XBF2  5231T XBF3  1212TENDALIGNEDAND COMPUTE THE ANGLES BETWEEN THESE VECTORS USING THE EUCLIDEANINNER PRODUCT AND IDENTIFY WHICH VECTORS ARE ORTHOGONALITEM SHOW THAT A SET OF NONZERO VECTORS P1P2LDOTSPM THAT  ARE MUTUALLY ORTHOGONAL SO THAT LA PIPJ RA  0 TEXT IF  I NEQ JIS LINEARLY INDEPENDENT  ORTHOGONALITY IMPLIES LINEAR INDEPENDENCE ITEM LET S BE A VECTOR SPACE WITH AN INDUCED NORM  SHOW THAT   LA XBFYBF RA   XBF   YBF  IF AND ONLY IF A XBF   B YBF  0 FOR SOME SCALARS A AND BEXSKIPSETEXSECTREFSECWIPITEM PERFORM THE SIMPLIFICATIONS TO GO FROM THE COMPARISON IN    REFEQDETECT1 TO THE COMPARISON IN REFEQDETECT2  ITEM SHOW BY INTEGRATION THAT INT11 FRAC1SQRT1T2 TNT TMT  BEGINCASES  PI  NM0   PI2  NM NNEQ 0   0  N NEQ MENDCASESHINT USE T  COS X IN THE INTEGRALEXSKIPSETEXSECTREFSECORTHOSUB    ITEM LABELEXORTHOGCOMP1 SHOW THAT THE ORTHOGONAL COMPLEMENT OF A SUBSPACE IS A SUBSPACEITEM LABELEXORTHOCOMP PROVE ITEMS 2 THROUGH 5 OF THEOREM  REFTHMORTHOGCOMP  HINT FOR ITEM 5 USE THEOREM REFTHMHDECOMPEXSKIPSETEXSECTREFSECLINTRANSITEM DETERMINE THE RANGE AND NULLSPACE OF THE FOLLOWING LINEAR  OPERATORS MATRICES A  BEGINBMATRIX10  54  2 4 ENDBMATRIXQQUADQQUADB  BEGINBMATRIX101  549  246 ENDBMATRIXITEM   LET X AND Y BE VECTOR SPACES OVER THE SAME SET OF  SCALARS  LET LTXY DENOTE THE SET OF ALL LINEAR TRANSFORMATIONS  FROM X TO Y  LET L AND M BE OPERATORS FROM LTXY  DEFINE AN ADDITION OPERATOR BETWEEN L AND M AS LMX  LX MXFOR ALL X IN X  ALSO DEFINE SCALAR MULTIPLICATION BY ALX  ALXSHOW THAT LTXYIS A LINEAR VECTOR SPACE ITEM LET X Y AND Z BE LINEAR VECTOR SPACES OVER THE SAME SET   OF SCALARS AND LET L1MC XRIGHTARROW Y AND L2MC Y   RIGHTARROW Z BE LINEAR OPERATORS SHOW THAT THE COMPOSITION   L2L1MC XRIGHTARROW Z IS A LINEAR OPERATOREXSKIPSETEXSECTREFSECISDSPITEM PROVE LEMMA REFLEMVWUNIQUE  ITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR INTERSECTION V CAP W IS A SUBSPACEITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR SUM VW IS A SUBSPACE ITEM SHOW THAT THE SET VOPLUS W HAS THE SAME ALGEBRAIC STRUCTURE   AS DOES THE SET V  W ESTABLISHING THAT THE ISOMORPHISM HOLDSITEM CITEP 200NAYLORSELL LET X  L2PIPI AND LET S1  LSPAN1COS TCOS 2TLDOTSQQUAD QQUAD S2 LSPANSIN T SIN 2T LDOTSBEGINENUMERATEITEM SHOW THAT S1 OPLUS S2 AND S1  S2 ARE ISOMORPHICITEM SHOW THAT DIMENSIONS1 OPLUS S2  DIMENSIONS1S2ENDENUMERATEITEM LABELEXORTHODIS SHOW THAT  BEGINENUMERATE  ITEM IF V AND W ARE ORTHOGONAL SUBSPACES THEN THEY ARE    DISJOINT  ITEM IF V AND W ARE DISJOINT THEY ARE NOT NECESSARILY    ORTHOGONAL  ENDENUMERATEITEM LET S BE A LINEAR SPACE AND ASSUME THAT S  S1  S2   CDOTS  SN WHERE THE SI ARE MUTUALLY DISJOINT LINEAR  SUBSPACES OF S  LET BI BE A HAMEL BASIS OF SI  SHOW THAT  B  B1 CUP B2CUP CDOTS CUP BN IS A HAMEL BASIS FOR SITEM PROVE THEOREM REFTHMVWISOITEM LET V AND W BE LINEAR SUBSPACES OF A FINITEDIMENSIONAL  LINEAR SPACE S  SHOW THAT DIMENSIONVW  DIMENSIONV  DIMENSIONW  DIMENSIONV CAPWTHEN CONCLUDE THAT DIMENSIONVOPLUS W  DIMENSIONV  DIMENSIONWEXSKIPSETEXSECTREFSECPROJECTIONSITEM IF VBF IS A VECTOR SHOW THAT THE MATRIX WHICH PROJECTS ONTO LSPANVBF IS PV  FRACVBF VBFHVBFHVBFITEM SHOW THAT THE MATRIX PA IN REFEQPROJMAT1 IS A PROJECTION  MATRIXITEM FOR THE PROJECTION MATRIX PAW INREFEQPROJMAT2  BEGINENUMERATE  ITEM SHOW THAT PAW2  PAW  ITEM SHOW THAT PAWPERP  IPAW IS ORTHOGONAL TO    PAW USING THE WEIGHTED INNER PRODUCT THAT IS    PAWH W PAWPERP  0  ENDENUMERATE  ITEM LET  PBF1  BEGINBMATRIX 123  4 ENDBMATRIX QQUADPBF2  BEGINBMATRIX 4  2  6  7 ENDBMATRIX QQUADPBF3  BEGINBMATRIX 3  4  2  1 ENDBMATRIXAND  XBF  BEGINBMATRIX 1  2  3  7 ENDBMATRIXDETERMINE THE NEAREST VECTOR XBFHAT INLSPANPBF1PBF2PBF3  ALSO DETERMINE THE ORTHOGONALCOMPLEMENT OF XBF IN LSPANPBF1PBF2PBF3ITEM LET A BE A MATRIX WHICH CAN BE FACTORED ASBEGINEQUATIONA  U SIGMA VHLABELEQSVDEXER1ENDEQUATIONSUCH THAT UH U  I QQUAD VH V  I AND SIGMA IS A DIAGONAL MATRIX WITH REAL VALUESTHE FACTORIZATION IN REFEQSVDEXER1 IS THE SINGULAR VALUEDECOMPOSITION SEE CHAPTER REFCHAPSVDSHOW THAT PA  PU  ITEM TWO ORTHOGONAL PROJECTION MATRICES PA AND PB ARE SAID TO  BE ORTHOGONAL IF PAPB  0  THIS IS DENOTED AS PA PERP PB  INDEXPROJECTION OPERATORSORTHOGONAL SHOW THAT  BEGINENUMERATE  ITEM PA AND PB ARE ORTHOGONAL IF AND ONLY IF THEIR RANGES ARE    ORTHOGONAL  ITEM PAPB IS A PROJECTION OPERATOR IF AND ONLY IF PA AND    PB  ARE ORTHOGONAL  ENDENUMERATEITEM SHOW THAT THE CBF WHICH MINIMIZES REFEQWPROJ1 IS CBF  AHWA1AHW XBFSO THAT THE WEIGHTED PROJECTION OF XBF IS ACBF AAHWA1AHW XBFITEM PROVE THEOREM REFTHMRNPITEM LET P1P2LDOTS PM BE A SET OF ORTHOGONAL PROJECTIONS WITH PIPJ  0 FOR I NEQ JSHOW THAT Q  P1  P2  CDOTS  PM IS AN ORTHOGONAL  PROJECTIONITEM LABELEXIMP IF P IS A PROJECTION OPERATOR SHOW THAT IP  IS A PROJECTION OPERATOR  DETERMINE THE RANGE AND NULLSPACE OF  IPITEM LET S BE A VECTOR SPACE AND LET V1V2LDOTS VN BE  LINEAR SUBSPACES SUCH THAT VI IS ORTHOGONAL FROM SUMJ NEQ I  VJ FOR EACH I AND WHERE S  V1  V2  CDOTS  VNLET PJ BE THE PROJECTION ON S FOR WHICH RANGEPJ  VJ ANDNULLSPACEPJ  SUMJ NEQ K VK  DEFINE AN OPERATOR P  LAMBDA1 P1  LAMBDA2 P2  CDOTS  LAMBDAN PNBEGINENUMERATEITEM SHOW THAT IF XBF IN VJ THEN P XBF  LAMBDAJ XBFITEM SHOW THAT P IS A PROJECTION IF AND ONLY IF LAMBDAJ IS  EITHER 0 OR 1ENDENUMERATEITEM LET A AND B BE MATRICES SUCH THAT AHB  0  THEN V   RANGEA AND W  RANGEB ARE ORTHOGONALBEGINENUMERATESHOW THAT PA  I  PB ITEM SHOW THAT A VECTOR XBF CAN BE DECOMPOSED AS  XBF  PA XBF  PBXBF  PA XBF  IPAXBF ENDENUMERATE ITEM SHOW THAT IF V AND W ARE CLOSED SUBSPACES OF A HILBERT   SPACE S AND IF V PERP W THEN VW IS A CLOSED SUBSPACE OF   S NS P 297EXSKIPSETEXSECTREFSECGRAMSCHMITITEM USING A SYMBOLIC MANIPULATION PACKAGE WRITE A FUNCTION WHICH  PERFORMS THE GRAMSCHMIDT ORTHOGONALIZATION OF A SET OF FUNCTIONSITEM DETERMINE THE FIRST FOUR POLYNOMIALS ORTHOGONAL OVER 01  ASYMBOLIC MANIPULATION PACKAGE IS RECOMMENDEDITEM FOR THE FUNCTIONS SHOWN IN FIGURE REFFIGGSEX DETERMINE A  SET OF ORTHOGONAL FUNCTIONS SPANNING THE SAME SPACE USING THE  FUNCTIONS IN THE ORDER SHOWN   SOMETHING LIKE THE PROAKIS EXERCISE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRGRAMSCHMIDT3      CAPTIONFUNCTIONS TO ORTHOGONALIZE      LABELFIGGSEX    ENDCENTER  ENDFIGUREITEM MODIFY ALGORITHM REFALGQR1 SO THAT IT ONLY RETAINS COLUMNS  OF Q THAT ARE NONZERO MAKING CORRESPONDING ADJUSTMENTS TO R  COMMENT ON THE PRODUCT QR IN THIS CASE GRAMSCHMIDT2MITEM MODIFY ALGORITHM REFALGQR1 TO COMPUTE A SET OF ORTHOGONAL  VECTORS WITH RESPECT TO THE WEIGHTED INNER PRODUCT LA XBF YBF  RA  XBFT W YBF FOR A POSITIVE DEFINITE SYMMETRIC MATRIX WITEM LABELEXMGS MODIFIED GRAMSCHMIDT INDEXMODIFIED    GRAMSCHMIDT INDEXGRAMSCHMIDTMODIFIED THE  COMPUTATIONS OF THE GRAMSCHMIDT ALGORITHM CAN BE REORGANIZED TO BE  MORE STABLE NUMERICALLY  IN THESE MODIFIED COMPUTATIONS A COLUMN  OF Q AND A EM ROW OF R IS PRODUCED AT EACH ITERATION  THE  REGULAR GRAMSCHMIDT PROCESS PRODUCES A COLUMN OF Q AND A COLUMN  OF R AT EACH ITERATION   LET THE KTH COLUMN OF Q BE DENOTED AS  QBFK AND LET THE KTH ROW OF R BE DENOTED AS RBFKT  BEGINENUMERATE  ITEM SHOW THAT FOR AN MATSIZEMN MATRIX A A  SUMI1K1 QBFI RBFIT  SUMIKN QBFIRBFIT  ZEROBF AKWHERE AK IS MATSIZEMNK1ITEM LET AK  ZBFK B WHERE B IS MATSIZEMNK  AND EXPLAIN WHY THE KTH COLUMN OF Q AND THE KTH ROW OF R ARE  GIVEN BY RKK   EBFK QQUAD QBFK  EBFKRKK QQUADRKK1LDOTSRKN  QBFKTBITEM THEN SHOW THAT THE NEXT ITERATION CAN BE STARTED BY COMPUTING A  SUMI1K QBFI RBFIT  ZEROBF AK1WHERE AK1  B  QBFKRKK1LDOTSRKNITEM CODE THE MODIFIED GRAMSCHMIDT ALGORITHM IN SC MATLAB  ENDENUMERATEENDEXERCISESSECTIONREFERENCESMUCH OF THE MATERIAL ON METRIC SPACES HILBERT SPACES AND BANACHSPACES PRESENTED HERE IS SIGNIFICANTLY COMPRESSED FROMCITENAYLORSELL  IN THEIR EXPANDED TREATMENT THEY PROVIDE PROOFS OFSEVERAL POINTS THAT WE HAVE MERELY MENTIONED  ANOTHER SOURCE ONVECTOR SPACES IS CITEFRIEDMAN  AN EXCELLENT HISTORICAL SOURCE ONVECTOR SPACES AND THEIR APPLICATIONS TO SIGNAL PROCESSING ANDENGINEERING IS CITELUENBERGER1969  FUNCTION SPACES WITH ANEMPHASIS ON SERIES REPRESENTATIONS ARE DISCUSSED IN CITEKEENERSIMILAR TREATMENT OF METRIC AND VECTOR SPACES IS CITEFRANKS  OURDISCUSSION OF THE MODIFIED GRAMSCHMIDT PROCESSINDEXGRAMSCHMIDTMODIFIED IS DRAWN FROM CITEGVLEXTENSIVE PROPERTIES OF THE ORTHOGONAL POLYNOMIALS INTRODUCED HERE AREDISCUSSED AND TABULATED IN CITEABRAMOWITZ SEE ALSO CITEWALTER1994AN EXTENSION OF THE CONCEPT OF A BASIS IS THAT OF A EM  FRAMEINDEXFRAME WHICH PROVIDES AN OVERDETERMINED SET OF  REPRESENTATIONAL FUNCTIONS  A TUTORIAL INTRODUCTION TO FRAMES WITH  APPLICATIONS IN SIGNAL PROCESSING APPEARS IN CITEPEI1997 LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERREPRESENTATION AND APPROXIMATION IN VECTOR SPACESLABELCHAPVECTAPBEGINQUOTESOURCEMICHAEL SPIVAKEM CALCULUS ON MANIFOLDS  ANY GOOD MATHEMATICAL COMMODITY IS WORTH GENERALIZINGENDQUOTESOURCESECTIONTHE APPROXIMATION PROBLEM IN HILBERT SPACELABELSECHILBAPPROXLET SCDOT BE A NORMED LINEAR VECTOR SPACE FOR SOME NORM CDOT   LET T   PBF1ALLOWBREAK PBF2ALLOWBREAK LDOTSALLOWBREAK PBFM SUBSET S BE A SET OFLINEARLY INDEPENDENT VECTORS IN A VECTOR SPACE S AND LET V LSPANT   THE ANALYSIS PROBLEM IS THIS GIVEN A VECTOR XBFIN S FINDTHE COEFFICIENTS C1C2LDOTSCM SO THATBEGINEQUATION XHAT  C1 PBF1  C2 PBF2   CDOTS CM PBFMLABELEQAPPROX1ENDEQUATIONAPPROXIMATES XBF AS CLOSELY AS POSSIBLE  THE HAT  CARETINDICATES THAT THIS IS OR MAY BE AN APPROXIMATIONINDEXAPPROXIMATION THAT IS WE WISH TO WRITEBEGINALIGNEDXBF  XBFHAT  EBF  C1 PBF1  C2 PBF2   CDOTS CM PBFM  EBFENDALIGNEDWHERE EBF IS THE APPROXIMATION ERROR SO THAT  XBF  XBFHAT   EBF IS AS SMALL AS POSSIBLE  THE PROBLEM IS DIAGRAMMED IN FIGUREREFFIGAPPROX1 FOR M2BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRORTHOGPROJ3    CAPTIONTHE APPROXIMATION PROBLEM    LABELFIGAPPROX1  ENDCENTERENDFIGUREOF COURSE IF XBF IN V THEN IT IS POSSIBLE TO FIND COEFFICIENTS SOTHAT  XBF  XBFHAT   0  THE PARTICULAR NORM CHOSEN INPERFORMING THE MINIMIZATION AFFECTS THE ANALYTIC APPROACH TO THEPROBLEM AND THE FINAL ANSWER  IF THE L1 OR L1 NORM IS CHOSENTHEN THE ANALYSIS INVOLVES ABSOLUTE VALUES WHICH MAKES AN ANALYTICALSOLUTION INVOLVING DERIVATIVES DIFFICULT  IF THE LINFTY ORLINFTY NORM IS CHOSEN THE ANALYSIS MAY INVOLVE DERIVATIVES OFTHE MAX FUNCTION WHICH IS ALSO DIFFICULT  THIS APPROXIMATIONPROBLEM IS DISCUSSED IN CHAPTER REFCHAPAPPROX  IF THE L2 OR L2 NORM IS CHOSEN MANY OF THE ANALYTICALDIFFICULTIES DISAPPEAR  THE NORM IS THE INDUCED NORM AND THEPROPERTIES OF THE PROJECTION THEOREM CAN BE USED TO FORMULATE THESOLUTION  ALTERNATIVELY THE SOLUTION CAN BE OBTAINED USING CALCULUSTECHNIQUES  ACTUALLY FOR PROBLEMS POSED USING THE LP NORMS AGENERALIZATION OF THE PROJECTION THEOREM CAN BE USED OPTIMIZING INBANACH SPACE RATHER THAN HILBERT SPACE BUT THIS LIES BEYOND THE SCOPEOF THIS BOOK  CHOOSING THE L2 NORM ALLOWS FAMILIAR EUCLIDEANGEOMETRY TO BE USED TO DEVELOP INSIGHT  THE APPROXIMATION PROBLEMWHEN THE INDUCED NORM IS USED FOR EXAMPLE EITHER AN L2 OR L2NORM IS KNOWN AS THE HILBERT SPACE APPROXIMATION PROBLEMTO DEVELOP GEOMETRIC INSIGHT INTO THE APPROXIMATION PROBLEM THEANALYSIS FORMULAS ARE PRESENTED BY STARTING WITH THE APPROXIMATIONPROBLEM WITH ONE ELEMENT IN T AIDED BY A KEY OBSERVATION THE ERRORIS ORTHOGONAL TO THE DATA  THE ANALYSIS IS THEN EXTENDED TO TWODIMENSIONS THEN TO ARBITRARY DIMENSIONS  WE WILL BEGIN FIRST WITHGEOMETRIC PLAUSIBILITY AND CALCULUS THEN PROVE THE RESULT USING THECAUCHYSCHWARZ INEQUALITYTO BEGIN LET T IN RBB2 CONSIST OF ONLY ONE VECTORTPBF1  FOR A VECTOR XBF IN RBB2 WE WISH TO REPRESENTXBF AS A LINEAR COMBINATION OF T XBF  C1 PBF1  EBFIN SUCH A WAY AS TO MINIMIZE THE NORM OF THE APPROXIMATION ERROR EBF  IN THIS SIMPLEST CASE THERE IS ONLY THE PARAMETER C1 TOIDENTIFY  THE SITUATION IS ILLUSTRATED IN FIGURE REFFIGAPPROX1AABEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODESUBFIGUREONE VECTOR IN TINPUTPICTUREDIRORTHOGPROJ4 QQUAD SUBFIGURETWO VECTORS IN T INPUTPICTUREDIRORTHOGPROJ5    CAPTIONAPPROXIMATION WITH ONE AND TWO VECTORS    LABELFIGAPPROX1A  ENDCENTERENDFIGUREIF THE L2 OR L2 NORM IS USED IT MAY BE OBSERVED GEOMETRICALLYTHAT BF THE ERROR IS MINIMIZED WHEN THE ERROR IS ORTHOGONAL TO VTHAT IS WHEN THE ERROR IS ORTHOGONAL TO THE DATA THAT FORMS OURESTIMATE  WRITTEN MATHEMATICALLY THE NORM OF THE ERROR  EBF IS MINIMIZED WHEN  EBF PERP PBF1 OR LA XBF  C1 PBF1 PBF1 RA  0USING THE PROPERTIES OF INNER PRODUCTSBEGINEQUATIONC1  FRAC LA XBFPBF1 RA  PBF1 22LABELEQNORM3ENDEQUATIONGEOMETRICALLY THE QUANTITY FRAC LA XBFPBF1 RA  PBF1 22IS THE BF PROJECTION OF THE VECTOR XBF IN THE DIRECTION OFPBF1 IT IS THE LENGTH OF THE SHADOW THAT XBF CASTS ONTOPBF1 EXPRESSED AS A PROPORTION OF THE LENGTH OF PBF1THE SAME APPROXIMATION FORMULA MAY ALSO BE OBTAINED BY CALCULUS  WEFIND C1 TO MINIMIZE  XBF  C1 PBF122  LA XBFC1PBF1XBFC1PBF1RABY TAKING THE DERIVATIVE WITH RESPECT TO C1 AND EQUATING THE RESULTTO ZERO  THIS GIVES THE SAME ANSWER AS REFEQNORM3CONTINUING OUR DEVELOPMENT WHEN T CONTAINS TWO VECTORS WE CAN WRITETHE APPROXIMATION AS XBF  C1 PBF1  C2 PBF2  EBFFIGURE REFFIGAPPROX1AB ILLUSTRATES THE CONCEPT FOR VECTORS INRBB3  IT IS CLEAR FROM THIS FIGURE THAT IF EUCLIDEANDISTANCE IS USED THE ERROR IS ORTHOGONAL TO THE DATA PBF1 ANDPBF2  THIS GIVES THE FOLLOWING ORTHOGONALITY CONDITIONS BEGINALIGNEDLA XBF  C1 PBF1  C2 PBF2PBF1 RA  0LA XBF  C1 PBF1  C2 PBF2PBF2 RA  0ENDALIGNEDEXPANDING THESE USING THE PROPERTIES OF INNER PRODUCTS GIVESBEGINALIGNED LA XBFPBF1 RA  C1 LA PBF1PBF1 RA  C2 LAPBF2PBF1 RA LA XBFPBF2 RA  C1 LA PBF1PBF2 RA  C2 LAPBF2PBF2 RAENDALIGNEDWHICH CAN BE WRITTEN MORE CONCISELY IN MATRIX FORMBEGINEQUATIONBEGINBMATRIX LA PBF1PBF1RA  LA PBF2PBF1RA LA PBF1PBF2RA  LA PBF2PBF2RAENDBMATRIXBEGINBMATRIX C1  C2 ENDBMATRIX BEGINBMATRIX LA XBFPBF1RA  LA XBFPBF2RA ENDBMATRIXLABELEQPROJ1ENDEQUATIONSOLUTION OF THIS MATRIX EQUATION PROVIDES THE DESIRED COEFFICIENTSBEGINEXAMPLE  SUPPOSE XBF  123T PBF1  110T AND PBF2   210T  IT IS CLEAR THAT XBFHAT  C1 PBF1  C2 PBF2 CANNOT BE AN EXACT REPRESENTATION OF XBF SINCE THERE IS NO WAY TOMATCH THE THIRD ELEMENT  USING REFEQPROJ1 WE OBTAINLEFTBEGINARRAYCC 2  3  3  5 ENDARRAYRIGHTLEFTBEGINARRAYCC C1  C2 ENDARRAYRIGHT LEFTBEGINARRAYC 3  4 ENDARRAYRIGHTTHIS CAN BE SOLVED TO GIVE C1   3 QUADQUAD C2  1 THEN THE APPROXIMATION VECTOR IS XBFHAT  C1 PBF1  C2 PBF2  3110T  210T 120T NOTE THAT THE APPROXIMATION FBFHAT IS THE SAME AS FBF IN THEFIRST TWO COEFFICIENTS  THE VECTOR HAS BEEN BF PROJECTEDINDEXPROJECTION ONTO THE PLANE FORMED BY THE VECTORS PBF1 ANDPBF2  THE ERROR IN THIS CASE HAS LENGTH 3ENDEXAMPLEJUMPING NOW TO HIGHER NUMBERS OF VECTORS WHAT WE CAN DO FOR TWOVECTORS IN T WE CAN DO FOR M INGREDIENT VECTORS  WE APPROXIMATEXBF AS XBF  SUMI1M CI PBFI  EBF  XBFHAT  EBFTO MINIMIZE  EBF   XBF  XBFHAT  IF THE NORM USED ISTHE L2 OR L2 NORM THIS IS THE EM LINEAR LEASTSQUARESINDEXLINEAR LEASTSQUARES PROBLEM  WHENEVER THE NORM MEASURING THEAPPROXIMATION ERROR  EBF IS INDUCED FROM AN INNER PRODUCT WECAN EXPRESS THE MINIMIZATION IN TERMS OF AN ORTHOGONALITY CONDITIONTHE MINIMUMNORM ERROR MUST BE ORTHOGONAL TO EACH VECTOR PJ LA XBF SUMI1M CI PBFI PJ RA  0 QQUADJ12LDOTS MTHIS GIVES US M EQUATIONS IN THE M UNKNOWNS WHICH MAY BE WRITTENASBEGINEQUATIONBEGINBMATRIXLA PBF1PBF1 RA  LA PBF2PBF1 RA  CDOTS  LAPBFMPBF1 RA  LA PBF1PBF2 RA  LA PBF2PBF2 RA  CDOTS  LAPBFMPBF2 RA  VDOTS  VDOTS LA PBF1PBFM RA  LA PBF2PBFM RA  CDOTS  LA PBFMPBFM RA ENDBMATRIXBEGINBMATRIXC1  C2  VDOTS  CM  ENDBMATRIX BEGINBMATRIXLA XBFPBF1 RA  LA XBFPBF2 RA VDOTS  LA XBFPBFM RA  ENDBMATRIXLABELEQPROJ2ENDEQUATIONWE DEFINE THE VECTORBEGINEQUATION PBF  BEGINBMATRIX LA XBF PBF1 RA LA XBF PBF2 RA VDOTS  LA XBF PBFM RA ENDBMATRIXLABELEQPBFENDEQUATIONAS THE EM CROSSCORRELATION VECTOR INDEXCROSSCORRELATION ANDBEGINEQUATIONCBF  BEGINBMATRIXC1  C2  VDOTS  CM ENDBMATRIX    LABELEQCBF  ENDEQUATIONAS THE VECTOR OF COEFFICIENTS  THEN REFEQPROJ2 CAN BE WRITTEN AS RCBF  PBFWHERE R IS THE MATRIX OF INNER PRODUCTS IN REFEQPROJ2EQUATIONS OF THIS FORM ARE KNOWN AS THE BF NORMAL EQUATIONSINDEXNORMAL EQUATIONS SINCE THE SOLUTION MINIMIZES THE SQUARE OFTHE ERROR IT IS KNOWN AS A EM LEASTSQUARE INDEXLEASTSQUARESOR EM MINIMUM MEANSQUARE INDEXMINIMUM MEANSQUARE SOLUTIONDEPENDING ON THE PARTICULAR INNER PRODUCT USEDSUBSECTIONTHE GRAMMIAN MATRIXLABELSECGRAMMIANTHE MATSIZEMM MATRIX BEGINEQUATIONR  BEGINBMATRIXLA PBF1PBF1 RA  LA PBF2PBF1 RA  CDOTS  LAPBFMPBF1 RA  LA PBF1PBF2 RA  LA PBF2PBF2 RA  CDOTS  LAPBFMPBF2 RA  VDOTS  VDOTS LA PBF1PBFM RA  LA PBF2PBFM RA  CDOTS  LAPBFMPBFM RA ENDBMATRIXLABELEQGRAMDEFENDEQUATIONIN THE LEFTHAND SIDE OF REFEQPROJ2 IS SAID TO BE THE BF  GRAMMIAN INDEXGRAMMIAN OF THE SET T  SINCE THE IJTHELEMENT OF THE MATRIX IS RIJ  LA PBFJPBFIRAIT FOLLOWS THAT THE GRAMMIAN IS A HERMITIAN SYMMETRIC MATRIX THAT IS RH  RWHERE H INDICATES CONJUGATETRANSPOSE  SOME IMPLICATIONS OF THEHERMITIAN STRUCTURE ARE EXAMINED IN SECTION REFSECDIAGONALSOLUTION OF REFEQPROJ2 REQUIRES THAT R BE INVERTIBLE  THEFOLLOWING THEOREM DETERMINES CONDITIONS UNDER WHICH R ISINVERTIBLE  RECALL THAT A MATRIX R FOR WHICH  XBFH R XBF  0FOR ANY NONZERO VECTOR XBF IS SAID TO BE POSITIVEDEFINITE SEEBOX REFBOXPD  INDEXPOSITIVEDEFINITE AN IMPORTANT ASPECT OFPOSITIVEDEFINITE MATRICES IS THAT THEY ARE ALWAYS INVERTIBLE  IF RIS SUCH THAT XBFH R XBF GEQ 0FOR ANY NONZERO VECTOR XBF THEN R IS SAID TO BE EM  POSITIVESEMIDEFINITE INDEXPOSITIVESEMIDEFINITE INPUTLINALGDIRPOSDEFTEXBEGINTHEOREM LABELTHMGRAMMPD  A GRAMMIAN MATRIX R IS ALWAYS POSITIVE SEMIDEFINITE THAT IS  XBFH R XBF GEQ 0 FOR ANY XBF IN CBBM  IT IS  POSITIVE DEFINITE IF AND ONLY IF THE VECTORS  PBF1PBF2LDOTSPBFM   ARE LINEARLY INDEPENDENT INDEXLINEARLY INDEPENDENTENDTHEOREMBEGINPROOF  LET YBF  Y1Y2LDOTSYMT BE AN ARBITRARY VECTOR  THENBEGINEQUATIONBEGINALIGNEDYBFH R YBF  SUMI1MSUMJ1M YBARI YJ LAPBFJPBFIRA  SUMI1MSUMJ1M LA YJ PBFJYI PBFI RALABELEQPR1  LEFTLANGLESUMJ1M YJ PBFJSUMI1M YI PBFIRIGHTRANGLE  LEFT SUMJ1M YJ PBFJRIGHT2 GEQ 0ENDALIGNEDENDEQUATIONHENCE R IS POSITIVE SEMIDEFINITEIF R IS NOT POSITIVE DEFINITE THEN THERE IS A NONZERO VECTOR YBF SUCHTHAT YBFH R YBF  0SO THAT BY REFEQPR1 SUMI1M YI PBFI  0THUS THE PBFI ARE LINEARLY DEPENDENT  CONVERSELY IF R IS POSITIVE DEFINITE THEN  YBFH R YBF  0FOR ALL NONZERO YBF AND BY REFEQPR1 SUMI1M YI PBFI NEQ 0THIS MEANS THAT THE  PBFI ARE LINEARLYINDEPENDENT INDEXLINEARLY INDEPENDENTENDPROOFAS A COROLLARY TO THIS THEOREM WE GET ANOTHER PROOF OF THECAUCHYSCHWARZ INEQUALITY  THE MATSIZE22 GRAMMIAN R  BEGINBMATRIXLA XXRA  LA XYRA  LA YXRA  LA YY RA ENDBMATRIXIS POSITIVE SEMIDEFINITE WHICH MEANS THAT ITS DETERMINANT ISNONNEGATIVE LA XXRA LA YY RA  LA XYRALA YXRA GEQ 0WHICH IS EQUIVALENT TO REFEQSW1  THE CONCEPT OF USING ORTHOGONALITY FOR THE EUCLIDEAN INNER PRODUCT TOFIND THE MINIMUM NORM SOLUTION GENERALIZES TO EM ANY INDUCED NORMAND ITS ASSOCIATED INNER PRODUCTIF THE SET OF VECTORS PBF1PBF2LDOTSPBFM ARE ORTHOGONALINDEXORTHOGONAL THEN THE GRAMMIAN IN REFEQGRAMDEF ISDIAGONAL SIGNIFICANTLY REDUCING THE AMOUNT OF COMPUTATION REQUIRED TOFIND THE COEFFICIENTS OF THE VECTOR REPRESENTATION  IN THIS CASE THECOEFFICIENTS ARE OBTAINED SIMPLY BYBEGINEQUATION CJ  FRAC LA XBFPBFJRA LA PBFJPBFJ RA LABELEQPROJ4ENDEQUATIONEACH COEFFICIENT USES THE SAME PROJECTION FORMULA THAT WAS USED INREFEQPROJ1 FOR A SINGLE DIMENSION  THE COEFFICIENTS CAN ALSO BEREADILY INTERPRETED FOR ORTHOGONAL VECTORS THE COEFFICIENT OF EACHVECTOR INDICATES THE STRENGTH OF THE VECTOR COMPONENT IN THE SIGNALREPRESENTATIONBEGINEXERCISESITEM LABELEXGRAMDET THERE IS A CONNECTION BETWEEN GRAMMIANS  INDEXGRAMMIAN AND LINEAR INDEPENDENCE INDEXLINEAR INDEPENDENCE  AS DEMONSTRATED IN THEOREM REFTHMGRAMMPD  WE WILL EXPLORE THIS  CONNECTION FURTHER IN THIS PROBLEM  LET PBF1PBF2LDOTSPBFN BE A SET OF VECTORS AND LET  US SUPPOSE THAT THE FIRST K1 VECTORS OF THIS SET HAVE PASSED A  TEST FOR LINEAR INDEPENDENCE  WE FORM EBFK  CK1K PBF1  CK2K PBF2  CDOTS C1KPBFK1  PBFK AND WANT TO KNOW IF EBFK IS EQUAL TO ZERO FOR ANY SET OFCOEFFICIENTS CBFK  CK1K CK2KLDOTSC1K1TIF SO THEN PBFK IS LINEARLY DEPENDENT  LET AK  PBF1PBF2LDOTSPBFKBE A DATA MATRIX AND LET RK  AKHAK BE THE CORRESPONDINGGRAMMIANBEGINENUMERATEITEM SHOW THAT THE SQUARED ERROR CAN BE WRITTEN ASBEGINEQUATION EBFKH EBFK  SIGMAK2  CBFHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFKLABELEQGRAMDET1ENDEQUATIONFOR SOME HBFK AND RKK  IDENTIFY HBFK AND RKKITEM DETERMINE THE MINIMUM VALUE OF SIGMAK2 BY MINIMIZING  REFEQGRAMDET1 WITH RESPECT TO CBFK SUBJECT TO THE  CONSTRAINT THAT THE LAST ELEMENT OF CBFK IS EQUAL TO 1  HINT  TAKE THE GRADIENT OF CBFHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFK  LAMBDACBFHDBF  1WHERE LAMBDA IS A LAGRANGE MULTIPLIER AND DBF 00LDOTS01T  SHOW THAT WE CAN WRITE THE CORRESPONDINGEQUATIONS ASBEGINEQUATIONBEGINBMATRIX RK1  HBFK  HBFKH  RKKENDBMATRIXCBFK  SIGMAK2 DBFLABELEQGRAMDET2ENDEQUATIONITEM SHOW THAT REFEQGRAMDET2 CAN BE MANIPULATED TO BECOME SIGMAK2  RKK  HBFKH RK11 HBFKTHE QUANTITY SIGMAK2 IS CALLED THE EM SCHUR COMPLEMENTINDEXSCHUR COMPLEMENT OFRK  IF SIGMAK20 THEN PBFK IS LINEARLY DEPENDENTENDENUMERATEENDEXERCISESSECTIONTHE ORTHOGONALITY PRINCIPLETHE BF ORTHOGONALITY PRINCIPLE INDEXORTHOGONALITY PRINCIPLE FORLEASTSQUARES LS OPTIMIZATION INTRODUCED IN SECTIONREFSECHILBAPPROX IS NOW FORMALIZEDBEGINTHEOREM  THE ORTHOGONALITY PRINCIPLE LET PBF1PBF2LDOTSPBFM BE DATA  VECTORS IN A VECTOR SPACE S  LET XBF BE ANY VECTOR IN  S  IN THE REPRESENTATION XBF  SUMI1M CI PBFI  EBF   XBFHAT  EBFTHE INDUCED NORM INDEXINDUCED NORM OF THE ERROR VECTOR EBFIS MINIMIZED WHEN THE ERROR EBF  XBFXBFHAT IS ORTHOGONAL TOEACH OF THE DATA VECTORS LA XBF  SUMI1M CI PBFIPBFJRA  0QQUADJ12LDOTSMENDTHEOREMBEGINPROOF ONE PROOF RELIES ON THE PROJECTION THEOREM THEOREM  REFTHMPROJ WITH THE OBSERVATION THAT V   LSPANPBF1PBF2LDOTSPBFM IS A SUBSPACE OF S  WE PRESENT A  MORE DIRECT PROOF USING THE CAUCHYSCHWARZ INEQUALITY    IN THE CASE THAT XBF IN LSPANPBF1PBF2LDOTSPBFM  THE ERROR IS ZERO AND HENCE IS ORTHOGONAL TO THE DATA VECTORS  THIS  CASE IS THEREFORE TRIVIAL AND IS EXCLUDED FROM WHAT FOLLOWS    IF XBF NOTIN LSPANPBF1PBF2LDOTSPBFM LET YBF  BE A FIXED VECTOR THAT IS ORTHOGONAL TO ALL OF THE DATA VECTORS LA YBFPBFIRA  0QQUAD I12LDOTSMSUCH THAT  XBF  SUMI1M AI PBFI  YBFFOR SOME SET OF COEFFICIENTS  A1A2LDOTSAM  LET EBF BE AVECTOR SATISFYINGBEGINEQUATION X  SUMI1M CI PBFI  EBFLABELEQPROVEORTHOG1ENDEQUATIONFOR SOME SET OF COEFFICIENTS  C1 C2LDOTS CM  THEN BY THECAUCHYSCHWARZ INEQUALITY INDEXCAUCHYSCHWARZ INEQUALITYBEGINALIGNAT2 EBF 2 YBF 2 GEQ LA EBFYBFRA2  QQUADTEXTCAUCHYSCHWARZ NOTAG  LA XBFYBF RA  LA SUMI1M CI PBFI YBFRA2  LA XBFYBFRA2  QQUAD TEXTORTHOGONALITY OF YBF NOTAGENDALIGNATTHE LOWER BOUND IS INDEPENDENT OF THE COEFFICIENTS CI ANDHENCE NO SET OF COEFFICIENTS CAN MAKE THE BOUND SMALLER  BY THEEQUALITY CONDITION FOR THE CAUCHYSCHWARZ INEQUALITY THE LOWER BOUNDIS ACHIEVED   IMPLYING THE MINIMUM EBF   WHEN EBF  ALPHA YBFFOR SOME SCALAR ALPHA  SINCE EBF MUST SATISFYREFEQPROVEORTHOG1 IT MUST BE THE CASE THAT EBF  YBF HENCE THEERROR IS ORTHOGONAL TO THE DATAENDPROOFWHEN CBF IS OBTAINED VIA THE PRINCIPLE OF ORTHOGONALITY THEOPTIMAL ESTIMATE XBFHAT  SUMI1M CI PBFIIS ALSO ORTHOGONAL TO THE ERROR EBF  XBF  XBFHAT SINCE IT IS A LINEARCOMBINATION OF THE DATA VECTORS PBFI  THUSBEGINEQUATIONLA XBFHAT EBF RA  0LABELEQXHATORTHOENDEQUATIONSUBSECTIONREPRESENTATIONS IN INFINITEDIMENSIONAL SPACEIF THERE ARE AN INFINITE NUMBER OF VECTORS IN T  PBF1PBF2LDOTS THEN THE REPRESENTATION XBFHAT  SUMI1INFTY CI PBFIMUST BE REGARDED WITH SOME DEGREE OF SUSPICION BECAUSE A LINEARCOMBINATION IS DEFINED TECHNICALLY ONLY IN TERMS OF A FINITE SUMTHE CONVERGENCE OF THIS INFINITE SUM MUST THEREFORE BE EXAMINEDCAREFULLY  HOWEVER IF T IS AN ORTHONORMAL SET THEN THEREPRESENTATION CAN BE SHOWN TO CONVERGESECTIONERROR MINIMIZATION VIA GRADIENTSLABELSECGRADMININDEXGRADIENT WHILE THE ORTHOGONALITY THEOREM IS USED PRINCIPALLYTHROUGHOUT THIS CHAPTER AS THE GEOMETRICAL BASIS FOR FINDING A MINIMUMERROR APPROXIMATION UNDER AN INDUCED NORM IT IS PEDAGOGICALLYWORTHWHILE TO CONSIDER ANOTHER APPROACH BASED ON GRADIENTS WHICHREAFFIRMS WHAT WE ALREADY KNOW BUT DEMONSTRATES THE USE OF SOME NEWTOOLSMINIMIZING  EBF 2 FOR THE INDUCED NORM IN XBF  SUMI1M CI PBFI  EBFREQUIRES MINIMIZING BEGINALIGNJCBF  LA XBF  SUMJ1M CJ PBFJ XBF  SUMI1M CIPBFI RA NONUMBER   LA XBFXBF RA  2 REAL LEFTSUMI1M CBARI LA   XBFPBFIRARIGHT  SUMI1M SUMJ1M CJ CBARI LA PBFJ PBF I RA LABELEQGRADMIN1BENDALIGNUSING THE VECTOR NOTATIONS DEFINED IN REFEQPBF REFEQCBFAND REFEQGRAMDEF WE CAN WRITE REFEQGRADMIN1B ASBEGINEQUATIONJCBF   XBF2  2 REALLEFTCBFH PBFRIGHT  CBFH RT CBFLABELEQGRADMIN2ENDEQUATIONSOME GRADIENT FORMULAS ARE DERIVED IN SECTION REFSECIMPGRAD INPARTICULAR THE FOLLOWING GRADIENT FORMULAS ARE DERIVED PARTIALDCBFBAR DBFH CBF  ZEROBF QQUADPARTIALDCBFBAR CBFH DBF  DBF QQUADPARTIALDCBFBAR REALCBFH DBF  FRAC12 DBF QQUADPARTIALDCBFBAR CBFH R CBF  R CBFTAKING THE GRADIENT OF REFEQGRADMIN2 USING THE LAST TWO OF THESEWE OBTAINBEGINEQUATION PARTIALDCBFBAR  LEFT XBF2  2 REALCBFH PBF  CBFH RCBFRIGHT  PBF  R CBFLABELEQGRADMIN1AENDEQUATIONEQUATING THIS RESULT TO ZERO WE OBTAIN RCBF  PBFGIVING US AGAIN THE NORMAL EQUATIONS INDEXNORMAL EQUATIONSTO DETERMINE WHETHER THE EXTREMUM WE HAVE OBTAINED BY THE GRADIENT ISIN FACT A MINIMUM WE COMPUTE THE GRADIENT A SECOND TIME  WE HAVE THEHESSIAN MATRIX INDEXHESSIAN MATRIX PARTIALDCBFBAR  R CBF  RWHICH IS A POSITIVESEMIDEFINITE MATRIX SO THE EXTREMUM MUST BE AMINIMUMRESTRICTING ATTENTION FOR THE MOMENT TO REAL VARIABLES CONSIDER THEPLOT OF THE NORM OF THE ERROR JCBF AS A FUNCTION OF THE VARIABLESC1 C2 LDOTS CM  SUCH A PLOT IS CALLED AN EM ERROR  SURFACE INDEXERROR SURFACE BECAUSE JCBF IS QUADRATIC INCBF AND R IS POSITIVE SEMIDEFINITE THE ERROR SURFACE IS APARABOLIC BOWL  FIGURE REFFIGERRORSURF1 ILLUSTRATES SUCH AN ERRORSURFACE FOR TWO VARIABLES C1 AND C2  BECAUSE OF ITS PARABOLICSHAPE ANY EXTREMUM MUST BE A MINIMUM AND IS IN FACT A GLOBALMINIMUMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE PLOTJSURF    EPSFIGFILEPICTUREDIRQUADERREPS    CAPTIONAN ERROR SURFACE FOR TWO VARIABLES    LABELFIGERRORSURF1  ENDCENTERENDFIGURESECTIONMATRIX REPRESENTATIONS OF LEASTSQUARES PROBLEMSLABELSECMATLSWHILE VECTOR SPACE METHODS APPLY TO BOTH INFINITE ANDFINITEDIMENSIONAL VECTORS SIGNALS THE NOTATIONAL POWER OF MATRICESCAN BE APPLIED WHEN THE BASIS VECTORS ARE FINITEDIMENSIONAL  THELINEAR COMBINATION OF THE FINITE SET OF VECTORSPBF1PBF2LDOTSPBFM CAN BE WRITTEN AS XBFHAT  SUMI1M CI PBFI   PBF1 PBF2 CDOTSPBFMBEGINBMATRIX C1  C2   VDOTS  CM ENDBMATRIXTHIS IS THE LINEAR COMBINATION OF THE COLUMNS OF THE MATRIX ADEFINED BY  A  BEGINBMATRIX PBF1  PBF2  CDOTS  PBFM ENDBMATRIXWHICH WE COMPUTE BY XBFHAT  ACBFTHE APPROXIMATION PROBLEM CAN BE STATED AS FOLLOWS   FBOXPARBOX09TEXTWIDTH BEGINQUOTE DETERMINE CBF TO MINIMIZE  EBF22 IN THE PROBLEM BEGINEQUATION XBF  A CBF  EBF  XBFHAT  EBF LABELEQMATLS ENDEQUATION ENDQUOTE BOXEDBEGINEQUATIONBOXEDRULE8EM0EM2EM   ADD A LITTLE SIZE TO THE BOXTEXT DETERMINE CBF TO MINIMIZE  EBF22 IN THE EQUATION XBF  A CBF  EBF  XBFHAT  EBF  LABELEQMATLSENDEQUATIONNOINDENT THE MINIMUM  EBF22  XBF  ACBF2 OCCURS WHEN EBFIS ORTHOGONAL TO EACH OF THE VECTORS LA XBF  ACBF PBFJRA  0QQUAD J12LDOTSMSTACKING THESE ORTHOGONALITY CONDITIONS WE OBTAIN BEGINBMATRIX PBF1H  PBF2H  VDOTS  PBFMHENDBMATRIXXBF  A CBF  ZEROBFRECOGNIZING THAT THE STACK OF VECTORS IS SIMPLY AH WE OBTAINBEGINEQUATION AHA CBF  AH XBFLABELEQLMAT9ENDEQUATIONTHE MATRIX AH A IS THE GRAMMIAN R AND THE VECTOR AH XBF ISTHE CROSSCORRELATION PBF  WE CAN WRITE REFEQLMAT9 ASBEGINEQUATIONR CBF  AH XBF  PBFLABELEQGAZENDEQUATIONTHESE EQUATIONS ARE THE NORMAL EQUATIONS INDEXNORMAL EQUATIONSTHEN THE OPTIMAL LEASTSQUARES COEFFICIENTS AREBEGINEQUATIONBOXEDCBF  AHA1 AH XBF  R1 PBFLABELEQLMAT0ENDEQUATIONBY THEOREM REFTHMGRAMMPD AHA IS POSITIVE DEFINITE IF THEPBF1LDOTSPBFM ARE LINEARLY INDEPENDENT  THE MATRIX AHA1 AH IS CALLED A EM PSEUDOINVERSE INDEXPSEUDOINVERSEOF A AND IS OFTEN DENOTED ADAGGER  MORE IS SAID ABOUTPSEUDOINVERSES IN SECTION REFSECPSINV  WHILEREFEQLMAT0 PROVIDES AN ANALYTICAL PRESCRIPTION FOR THE OPTIMALCOEFFICIENTS IT SHOULD RARELY BE COMPUTED EXPLICITLY AS SHOWN SINCEMANY PROBLEMS ARE NUMERICALLY UNSTABLE SUBJECT TO AMPLIFICATION OFROUNDOFF ERRORS  NUMERICAL STABILITY IS DISCUSSED IN SECTIONREFSECMATCOND  STABLE METHODS FOR COMPUTING PSEUDOINVERSES AREDISCUSSED IN SECTIONS REFSECQR AND REFSECPSEUDOINVERSESVDIN SC MATLAB THE PSEUDOINVERSE MAY BE COMPUTED USING THE COMMANDTT PINVUSING REFEQLMAT0 THE APPROXIMATION ISBEGINEQUATIONBOXED XBFHAT  A CBF  AAHA1 AH XBFLABELEQLSMAT1ENDEQUATIONTHE MATRIX P  AAHA1AH IS A EM PROJECTION MATRIXINDEXPROJECTION MATRIX WHICH WE ENCOUNTERED IN SECTIONREFSECPROJECTIONS  THE MATRIX P PROJECTS ONTO THE RANGE OF ACONSIDER GEOMETRICALLY WHAT IS TAKING PLACE WE WISH TO SOLVE THEEQUATION A CBF  XBF BUT THERE IS NO EXACT SOLUTION SINCE XBFIS NOT IN THE RANGE OF A  SO WE PROJECT XBF ORTHOGONALLY DOWNONTO THE RANGE OF A AND FIND THE BEST SOLUTION IN THAT RANGE SPACETHE IDEA IS SHOWN IN FIGURE REFFIGPSOLBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRORTHOGPROJ6    CAPTIONPROJECTION SOLUTION    LABELFIGPSOL  ENDCENTERENDFIGUREA USEFUL REPRESENTATION OF THE GRAMMIAN R  AHA CAN BE OBTAINED BYCONSIDERING A AS A STACK OF ROWSBEGINEQUATIONA  BEGINBMATRIX QBF1H  QBF2H  VDOTS  QBFNHENDBMATRIXLABELEQXSTACKROWENDEQUATIONSO THAT AH  QBF1 QBF2LDOTSQBFN ANDBEGINEQUATIONAHA  BEGINBMATRIXQBF1  QBF2  CDOTS  QBFN ENDBMATRIXBEGINBMATRIXQBF1H  QBF2H  VDOTS  QBFNHENDBMATRIX  SUMI1N QBFI QBFIHLABELEQXSTACKROWENDEQUATIONSUBSECTIONWEIGHTED LEASTSQUARESLABELSECWLSINDEXWEIGHTED LEASTSQUARESA WEIGHT CAN ALSO BE APPLIED TO THE DATA POINTS REFLECTING THECONFIDENCE IN THE DATA AS ILLUSTRATED BY THE NEXT EXAMPLE  THIS ISNATURALLY INCORPORATED INTO THE INNER PRODUCTDEFINE A WEIGHTED INNER PRODUCT AS LA XBFYBFRAW  XBFH W YBFTHEN MINIMIZING EBFW2  ACBF  XBFW2 LEADS TO THEWEIGHTED NORMAL EQUATIONSBEGINEQUATIONAH WACBF  AH WXBFLABELEQWLS1ENDEQUATIONSO THE COEFFICIENTS WHICH MINIMIZE THE WEIGHTED SQUARED ERROR AREBEGINEQUATION  LABELEQWLS2  CBF  AH W A1 AH W YBFENDEQUATIONANOTHER APPROACH TO REFEQWLS2 IS TO PRESUME THAT WE  HAVE AFACTORIZATION OF THE WEIGHT W  SHS SEE SECTIONREFSECCHOLESKY  THEN WE WEIGHT THE EQUATION  SA CBF APPROX SYBFMULTIPLYING THROUGH BY SAH AND SOLVING FOR CBF WE OBTAIN CBF  SAHSA1SAH SYBFWHICH IS EQUIVALENT TO REFEQWLS2SUBSECTIONSTATISTICAL PROPERTIES OF THE LEASTSQUARES ESTIMATELABELSECLSPROPSUPPOSE THAT THE SIGNAL XBF HAS THE TRUE MODEL ACCORDING TO THEEQUATIONBEGINEQUATIONXBF  A CBF0  EBFLABELEQGRAMSTATENDEQUATIONFOR SOME TRUE MODEL PARAMETER VECTOR CBF0 AND THAT WEASSUME A STATISTICAL MODEL FOR THE MODEL ERROR EBF ASSUME THATEACH COMPONENT OF EBF IS A ZEROMEAN IID RANDOM VARIABLE WITHVARIANCE SIGMAE2  THE ESTIMATED PARAMETER VECTOR ISBEGINEQUATIONCBF  AHA1 AH XBFLABELEQCESTENDEQUATIONTHIS LEASTSQUARES ESTIMATE BEING A FUNCTION OF THE RANDOM VECTORXBF IS ITSELF A RANDOM VECTOR  WE WILL DETERMINE THE MEAN ANDCOVARIANCE MATRIX FOR THIS RANDOM VECTORBEGINDESCRIPTIONITEMMEAN OF CBF SUBSTITUTING THE TRUE MODEL OF REFEQGRAMSTAT INTOREFEQCEST WE OBTAINBEGINALIGNEDCBF   AHA1 AHA CBF0  AHA1AH  EBF  CBF0  AHA1AH  EBFENDALIGNEDIF WE NOW TAKE THE EXPECTED VALUE OF OUR ESTIMATED PARAMETER VECTOR WEOBTAIN ECBF  ECBF0  AHA1AH  EBF  CBF0SINCE EACH COMPONENT OF EBF HAS ZERO MEAN  THUS THE EXPECTEDVALUE OF THE ESTIMATE IS EQUAL TO THE TRUE VALUE  SUCH AN ESTIMATE ISSAID TO BE BF UNBIASED INDEXUNBIASEDITEMCOVARIANCE OF CBF THE COVARIANCE CAN BE WRITTEN ASBEGINALIGNEDCOVCBF  ECBF  CBF0CBF  CBF0H  AHA1 AH EEBF EBFH AAHA1ENDALIGNEDSINCE THE COMPONENTS OF EBF ARE IID IT FOLLOWS THATEEBF EBFH  SIGMAE2 I SO THAT COVCBF  SIGMAE2AHA1  SIGMAE2 R1ITEM SMALLEST COVARIANCE ANOTHER INTERESTING FACT OF ALL POSSIBLE  UNBIASED LINEAR ESTIMATES THE ESTIMATOR REFEQLMAT0 HAS THE  SMALLEST COVARIANCE  SUPPOSE WE HAVE ANOTHER UNBIASED LINEAR  ESTIMATOR CBFTILDE GIVEN BY CBFTILDE  L XBFWHERE ECBFTILDE  CBF0  USING OUR STATISTICAL MODEL REFEQGRAMSTAT WE OBTAIN CBFTILDE  LA CBF0  L EBFIN ORDER FOR THE ESTIMATE CBFTILDE TO BE UNBIASED WE MUST HAVEECBFTILDE  CBF0 SO LA  IWE THEREFORE OBTAIN CBFTILDE  CBF0  L EBF  THE COVARIANCE OFCBFTILDE IS COVCBFTILDE  ECBFTILDE  CBF0CBFTILDE  CBF0H SIGMAE2 LLHWE WILL SHOW THAT LLH  R1 IN THE SENSE THAT THE MATRIX LLH R1 IS POSITIVE SEMIDEFINITE  INDEXPOSITIVESEMIDEFINITE LET Z  L  R1AHTHEN FOR ANY ZBF  0 LEQ  ZH ZBF 2  LA ZH ZBF ZH ZBF RA  ZBFH Z ZHZBFBUT  ZZH  LLH  R1WHERE WE HAVE USED THE FACT THAT LA  I  THUS FOR ANY ZBF  ZBFH LLH  R1 ZBF GEQ 0SO LLH  R1 IS POSITIVE SEMIDEFINITE OR R1 IS A SMALLERCOVARIANCE MATRIX  THE ESTIMATOR CBF IS SAID TO BE A BEST LINEARUNBIASED ESTIMATOR BLUE INDEXBEST LINEAR UNBIASED ESTIMATE  BLUE INDEXMINIMUM VARIANCE ESTIMATEIT WILL BE SHOWN IN CHAPTER REFCHAPEST THAT UNDER THECONDITION THAT THE NOISE EBF IS GAUSSIAN THE COVARIANCE OFCBF IS IN FACT THE SMALLEST COVARIANCE AMONG ALL POSSIBLEUNBIASED ESTIMATORS  WE SHALL SEE IN SECTION REFSECCRLB THATTHERE IS A LOWER BOUND ON THE VARIANCE OF UNBIASED ESTIMATORS INDEXCRAMERRAO LOWER BOUND CRLB ENDDESCRIPTIONSECTIONMINIMUM ERROR IN VECTOR SPACE APPROXIMATIONSLABELSECMINERRINDEXMINIMUM ERRORIN THIS SECTION WE EXAMINE HOW MUCH ERROR IS LEFT WHEN AN OPTIMALMINIMALNORM SOLUTION IS OBTAINED  UNDER THE MODEL THAT XBF  SUMI1M CI PBFI  EBFWHEN THE COEFFICIENTS ARE FOUND SO THAT THE ESTIMATION ERROR ISORTHOGONAL TO THE DATA WE HAVE XBF  XBFHAT  EBFMINWHERE EBFMIN DENOTES THE MINIMUM ACHIEVABLE ERRORTAKING THE SQUARED NORM OF BOTH SIDES WE OBTAINBEGINEQUATION  XBF 2   XBFHAT 2   EBFMIN 2LABELEQEMIN1ENDEQUATIONTHIS RESULT SOMETIMES CALLED THE STATISTICIANS PYTHAGOREAN THEOREMINDEXPYTHAGOREAN THEOREMSTATISTICIANSFOLLOWS BECAUSE XBFHAT IS ORTHOGONAL TO THE MINIMUMNORM ERROR LA XBFHAT EBFMINRA  0THE STATISTICIANS PYTHAGOREAN THEOREM IS ILLUSTRATED IN FIGUREREFFIGPYTHAG1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPYTHAG1    CAPTIONSTATISTICIANS PYTHAGOREAN THEOREM    LABELFIGPYTHAG1  ENDCENTERENDFIGURESEE ALSO LEMMA REFLEMPYTH  THE SQUARED NORM OF THE MINIMUMERROR IS  EBFMIN2   XBF2   XBFHAT2WHEN WE USE THE MATRIX FORMULATION WE CAN OBTAIN A MORE EXPLICITREPRESENTATION FOR THE MINIMUM ERROR  THEN XBFHAT  ACBF SOBEGINEQUATION  XBFHAT 2  CBFH AH A CBF  CBFH R CBF CBFH PBFLABELEQEMIN2ENDEQUATIONWHERE PBF FROM REFEQGAZ HAS BEEN EMPLOYED  THIS GIVES  EBFMIN2  XBFH XBF  CBFH PBFANOTHER FORM FOR XBFHAT2 IS OBTAINED FROMREFEQLSMAT1BEGINEQUATION XBFHAT2  ACBFH ACBF  XBFH AAHA1 AH XBFLABELEQEMIN3ENDEQUATIONTHENBEGINALIGNEDEBFMIN2   XBFH XBF  XBFH AAHA1 AH XBF  XBFHI  AAHA1AH XBFENDALIGNEDIT CAN BE SHOWN SEE EXERCISE REFEXREDUCEERR THAT BEGINEQUATION I  AAHA1AHLABELEQREDUCERRENDEQUATIONIS A POSITIVESEMIDEFINITE MATRIX INDEXPOSITIVESEMIDEFINITE MATRIXFROM WHICH WE CAN CONCLUDE THAT EBFMIN2 IS SMALLER THANXBF2BEGINEXERCISESITEM SHOW THAT REFEQXSTACKROW IS TRUE  ITEM SHOW THAT I  AAHA1AHIS POSITIVESEMIDEFINITE AND HENCE THAT THE MINIMUM ERROREBFMIN HAS SMALLER NORM THAN THE ORIGINAL VECTOR XBF  HINTCONSIDER 0 LEQ  BXBF2 WHERE B  I  AAHA1AHENDEXERCISESCHAPTERPARTAPPLICATIONS OF THE ORTHOGONALITY THEOREMLABELSECOTAPP1BECAUSE A NUMBER OF VECTOR SPACES AND INNER PRODUCTS CAN BEFORMULATED THE ORTHOGONALITY PRINCIPLE IS USED IN A VARIETY OFAPPLICATIONS  THE ORTHOGONALITY THEOREM PROVIDES THE FOUNDATION FOR AGOOD PART OF SIGNAL PROCESSING THEORY SINCE IT PROVIDES APRESCRIPTION FOR AN OPTIMUM ESTIMATOR BF IN THE OPTIMUM  LEASTSQUARES ESTIMATOR THE ERROR IS ORTHOGONAL TO THE DATA THETHEOREM IS APPLIED BY DEFINING AN INNER PRODUCT AND HENCE THE INDUCEDNORM TO MATCH THE NEEDS OF THE PROBLEM  UNDER VARIOUS DEFINITIONS OFINNER PRODUCTS MUCH OF APPROXIMATION THEORY ESTIMATION THEORY ANDPREDICTION THEORY CAN BE ACCOMMODATED  EXAMPLES ARE GIVEN IN THE NEXTSEVERAL SECTIONSSECTIONAPPROXIMATION BY CONTINUOUS POLYNOMIALSLABELSECPOLYAPPROX1INDEXPOLYNOMIAL APPROXIMATIONCONTINUOUS POLYNOMIALSSUPPOSE WE WANT TO FIND THE BEST POLYNOMIAL APPROXIMATION OF A REALCONTINUOUS FUNCTION FT OVER AN INTERVAL T IN AB IN THESENSE THAT INTAB FT PT2 DTIS MINIMIZED FOR A POLYNOMIAL PT OF DEGREE M1  THE VECTOR SPACEUNDERLYING THE PROBLEM IS S  CAB  WE WILL NAIVELY TAKE ASBASIS VECTORS THE FUNCTIONS 1TT2LDOTSTM1 SO THAT PT  C0  C1 T  C2 T2  CDOTS  CM1 TM1THE OPTIMAL COEFFICIENTS CAN BE DETERMINED FOR EXAMPLEDIRECTLY BY CALCULUS BUT THE ORTHOGONALITY THEOREM APPLIES USING THEINNER PRODUCT LA FG RA  INTAB FTGTDTTHEN USING REFEQPROJ2 WE OBTAINBEGINEQUATIONBEGINBMATRIXLA 11 RA  LA 1T RA  CDOTS  LA 1TM1 RA  LA T1 RA  LA TT RA  CDOTS  LA TTM1 RA VDOTS LA TM11 RA  LA TM1TRA  CDOTS  LA TM1TM1 RAENDBMATRIXBEGINBMATRIXC0  C1  VDOTS  CM ENDBMATRIX BEGINBMATRIX LA F1 RA  LA FT RA  VDOTS  LA FTM1  RA ENDBMATRIXLABELEQPOLYAPPROX1ENDEQUATIONIF WE TAKE THE SPECIFIC CASE THAT THE FUNCTION IS TO BE APPROXIMATEDOVER THE INTERVAL 01 THEN THE GRAMMIAN MATRIX INREFEQPOLYAPPROX1 CAN BE COMPUTED EXPLICITLY AS LA TITJ RA  INT01 TIJDT  FRAC1IJ1QQUADIJ01LDOTSM1 SO THATBEGINEQUATIONR  BEGINBMATRIX1  FRAC12  FRAC13  CDOTS   FRAC1M  FRAC12  FRAC13  FRAC14  CDOTS  FRAC1M1 VDOTS FRAC1M  FRAC1M1  FRAC1M2  CDOTS  FRAC12M ENDBMATRIXLABELEQHILBERTGENDEQUATIONA MATRIX OF THIS PARTICULAR FORM IS KNOWN AS A BF HILBERT MATRIXINDEXHILBERT MATRIX THE HILBERT MATRIX IS FAMOUS AS A CLASSICEXAMPLE OF A MATRIX THAT IS ILLCONDITIONED AS M INCREASES THEMATRIX BECOMES ILLCONDITIONED INDEXILLCONDITIONED EXPONENTIALLYFAST WHICH MEANS AS DISCUSSED IN SECTION REFSECMATCOND THAT ITWILL SUFFER FROM SEVERE NUMERICAL PROBLEMS IF M IS EVEN MODERATELYLARGE NO MATTER HOW IT IS INVERTED  BECAUSE OF THIS THE PARTICULARSET OF BASIS FUNCTIONS CHOSEN IS NOT RECOMMENDED  THE USE OF THELEGENDRE POLYNOMIALS DESCRIBED IN EXAMPLE REFEXMLEGENDREPOLY OROTHER ORTHOGONAL POLYNOMIALS IS PREFERRED FOR POLYNOMIALAPPROXIMATION BEGINEXAMPLE  LET FT  ET AND M3  FOR ONLY THREE PARAMETERS THE HILBERT  MATRIX REFEQHILBERTG IS STILL WELLCONDITIONED  THE VECTOR  ON THE RIGHTHAND OF REFEQPOLYAPPROX1 IS BBF  BEGINBMATRIX E1  1  E2 ENDBMATRIXAND THE COEFFICIENTS IN REFEQPOLYAPPROX1 ARE COMPUTED AS BEGINBMATRIXC0  C1  C2 ENDBMATRIX  R1 BBF BEGINBMATRIX 10130     08511     08392 ENDBMATRIXTHE APPROXIMATING POLYNOMIAL IS ET APPROX 10130  8511 T  8392 T2FIGURE REFFIGHILB1 SHOWS THE ABSOLUTE ERROR ET  PT FOR THISPOLYNOMIAL FOR T IN01  FOR COMPARISON THE ERROR WE WOULD GETBY APPROXIMATING ET BY THE FIRST THREE TERMS OF THE TAYLOR SERIESEXPANSION  ET APPROX 1  T  T22IS ALSO SHOWN AS IS THE WEIGHTED LEASTSQUARES WLS APPROXIMATIONDISCUSSED SUBSEQUENTLY  THE ERROR IN THE TAYLOR SERIES STARTS SMALLBUT INCREASES TO A LARGER VALUE THAN DOES THE LEASTSQUARESAPPROXIMATION  HOW WOULD THE TAYLOR SERIES HAVE COMPARED IF THESERIES HAD BEEN EXPANDED ABOUT THE MIDPOINT OF THE REGION ATT0FRAC12 USE THE FILE PROGSHILB1M THEN MOVE THE LEGEND AND SAVEBEGINFIGUREHTBP  CENTERLINEPSFIGFILEPICTUREDIRHILB1EPS  CAPTIONCOMPARISON OF LS WLS AND    TAYLOR SERIES APPROXIMATIONS TO ET   LABELFIGHILB1ENDFIGUREENDEXAMPLETHE BASIS FUNCTIONS OF THE PREVIOUS EXAMPLE GIVE RISE TO THE HILBERTMATRIX AS THE GRAMMIAN  HOWEVER A SET OF EM ORTHOGONALPOLYNOMIALS CAN BE USED THAT HAS A DIAGONAL AND HENCEWELLCONDITIONED GRAMMIAN  NOW SUPPOSE THAT FOR SOME REASON IT IS MORE IMPORTANT TO GET THEAPPROXIMATION MORE CORRECT ON THE EXTREMES OF THE INTERVAL OFAPPROXIMATION  WE WILL DENOTE THE APPROXIMATING POLYNOMIAL IN THISCASE BY PWT  TO ATTEMPT TO MAKE THE APPROXIMATION MORE EXACT ONTHE EXTREMES OF THE INTERVAL OF APPROXIMATION WE USE A WEIGHTED NORM INTAB WTFT  PWT2 DTWHICH IS INDUCED FROM THE INNER PRODUCT LA FG RA  INTAB SQRTWT FTGTDTBEGINEXAMPLE  CONTINUING THE EXAMPLE ABOVE WITH FT  ET OVER 01 TAKE  THE WEIGHTING FUNCTION AS WT  10T052THEN THE GRAMMIAN MATRIX IS R  BEGINPMATRIXFRAC12SQRT52  FRAC14SQRT52  FRAC316SQRT52FRAC14SQRT52  FRAC316SQRT52 FRAC532SQRT52FRAC316SQRT52 FRAC532SQRT52FRAC1396SQRT52ENDPMATRIXAND THE RIGHTHAND VECTOR COMPUTED NUMERICALLY IS BBF   138603    0860513   0690724THE APPROXIMATING POLYNOMIAL IS NOW PWT  10109  8535 T  8415 T2FIGURE REFFIGHILB1 SHOWS THE ERROR ET  PWT AND ET PT  AS EXPECTED THE ERROR IS SMALLER THOUGH ONLY SLIGHTLY FORPWT NEAR THE ENDPOINTS BUT LARGER IN BETWEENENDEXAMPLEAS VARIOUS WEIGHTINGS ARE IMPOSED THE ERROR AT SOME VALUES OF T ISREDUCED WHILE ERROR FOR OTHER VALUES OF T MAY INCREASE  THISRAISES THE FOLLOWING INTERESTING AND IMPORTANT QUESTION IS THERESOME WAY TO DESIGN THE APPROXIMATION SO THAT THE MAXIMUM ERROR ISMINIMIZED  THIS IS WHAT LINFTY APPROXIMATION IS ALL ABOUT  THEAPPROXIMATION IS FOUND SO THAT THE MAXIMUM ERROR IS MINIMIZED MORE WILL BE SAID ABOUT THIS IN CHAPTER REFCHAPAPPROXSECTIONAPPROXIMATION BY DISCRETE POLYNOMIALSLABELSECPOLYAPPROX2INDEXPOLYNOMIAL APPROXIMATIONDISCRETE POLYNOMIALSWE CAN APPROXIMATE DISCRETE SAMPLED DATA USING POLYNOMIALS IN AMANNER SIMILAR TO THE CONTINUOUS POLYNOMIAL APPROXIMATIONS OF SECTIONREFSECPOLYAPPROX1 USING A SET OF DISCRETETIME BASIS FUNCTIONS1KLDOTSKM1  WE DESIRE TO FIT AN M1ST ORDERPOLYNOMIAL THROUGH THE DATA POINTS X1X2LDOTSXN SO THATXK APPROX PKQQUAD K12LDOTSNWHERE PK  C0  C1 K  C2 K2  CDOTS  CM1 KM1THE POLYNOMIAL PK CAN BE WRITTEN AS PK  1 K K2 CDOTS KM1BEGINBMATRIXC0  C1  C2   VDOTS  CM1 ENDBMATRIXIF MN AND THE XK ARE DISTINCT THEN THERE EXISTS A POLYNOMIALTHE EM INTERPOLATING POLYNOMIAL INDEXINTERPOLATING POLYNOMIALPASSING EXACTLY THROUGH ALL N POINTS  IF M N THEN THERE ISPROBABLY NOT A POLYNOMIAL THAT WILL PASS THROUGH ALL N POINTS INWHICH CASE WE DESIRE TO FIND THE POLYNOMIAL TO MINIMIZE THE SQUAREDERROR SUMK1N XK  PK2THIS CAN BE EXPRESSED AS A VECTOR NORM  XBF  PBF 2WHICH IS INDUCED FROM THE EUCLIDEAN INNER PRODUCT LA XBF YBF RA XBFH YBF WHERE XBF  BEGINBMATRIX X1  X2  VDOTS XNENDBMATRIXQQUADTEXTANDQQUADPBF  BEGINBMATRIX P1  P2  VDOTS  PN ENDBMATRIXWE CAN WRITE PBF IN TERMS OF THE COEFFICIENTS OF THE POLYNOMIAL AS PBF  BEGINBMATRIX1  1  1 CDOTS  1 1  2  4  CDOTS  2M1 1  3  9  CDOTS  3M1 VDOTS 1  N  N2  CDOTS  NM1 ENDBMATRIXBEGINBMATRIXC0  C1  C2  VDOTS  CM1 ENDBMATRIX PBF1  PBF2 PBF3  CDOTS PBFMBEGINBMATRIXC0  C1   C2  VDOTS  CM1 ENDBMATRIX  P ABFTHE VECTORS PBFI I 12LDOTS M REPRESENT THE DATA IN THISAPPROXIMATION PROBLEM  IF P IS SQUARE IT IS CALLED A EM  VANDERMONDE MATRIX INDEXVANDERMONDE MATRIX ABOUT WHICH MORE ISPRESENTED IN SECTION REFSECVANDERMONDE  AS WITH THECONTINUOUSTIME POLYNOMIAL APPROXIMATION THERE MAY BE BETTER BASISFUNCTIONS FOR THIS PROBLEM FROM A NUMERICAL POINT OF VIEWUSING THIS NOTATION THE APPROXIMATION PROBLEM BECOMES XBF  P CBF  EBFWHICH IS A PROBLEM IN THE SAME FORM AS REFEQNORM3 FROM WHICHOBSERVE THAT THE CBF WHICH MINIMIZES  EBF2 IS CBF  PT P1 PT XBFTHE APPROXIMATED VECTOR PBF IS THUS PBF  P CBF  PPTP1PT XBFBEGINEXAMPLE WE DESIRE TO APPROXIMATE THE FUNCTION XK  SINKPI7USING A QUADRATIC POLYNOMIAL M3 TO OBTAIN THE BEST MATCH FORK1MC 7  THE P MATRIX ISBEGINBMATRIX1  1  1 1  2  4 1  3  9 1  4  16 1  5  25 1  6  36 1  7  49 ENDBMATRIXAND THE COEFFICIENTS ARE COMPUTED AS CBFT 006120588500833  FIGURE REFFIGDISCAPPROXA SHOWSXK AND FIGURE REFFIGDISCAPPROXB SHOWS THE ERROR PK XKBEGINFIGUREHTBP  CENTERLINEMBOXSUBFIGUREMBOXXKEPSFIGFILEPICTUREDIRDISCAPPROX1EPS           WIDTH045TEXTWIDTHQUAD     SUBFIGUREMBOXXK PKEPSFIGFILEPICTUREDIRDISCAPPROX2EPS           WIDTH045TEXTWIDTH  CAPTIONA DISCRETE FUNCTION AND THE ERROR IN ITS APPROXIMATION  LABELFIGDISCAPPROX USE DISCAPPROXMENDFIGUREENDEXAMPLESECTIONLINEAR REGRESSIONLABELSECLINREGFROM THE  DATA IN FIGURE REFFIGREGRESS1A WHERE THERE ARE NPOINTS XBFI I12LDOTSN WITH EACH XBFI  XIYIT ITWOULD APPEAR THAT WE CAN APPROXIMATELY FIT A LINEOF THE FORMBEGINEQUATIONYI APPROX A XI  B QQUAD I12LDOTSNLABELEQREGRESS1ENDEQUATIONFOR SUITABLY CHOSEN SLOPE A AND INTERCEPT B  AS STATED THIS IS AEM LINEAR REGRESSION INDEXREGRESSION PROBLEM THAT IS A PROBLEMOF DETERMINING A FUNCTIONAL RELATION BETWEEN THE MEASURED VARIABLESXI AND YI  NONLINEAR REGRESSIONS ARE ALSO USED SUCH AS THEQUADRATIC REGRESSIONBEGINEQUATION YI APPROX A0  A1 XI  A2 XI2LABELEQREGRESS2ENDEQUATIONOR WE MAY HAVE DATA VECTORS XBFI IN RBB3 WITH XBFI XIYIZIT AND WE MAY REGRESS AMONG THE POINTS ASBEGINEQUATIONZI APPROX A XI  B YI  C LABELEQREGRESS3ENDEQUATIONIN ALL SUCH REGRESSION PROBLEMS WE DESIRE TO CHOOSE THE REGRESSIONPARAMETERS SO THAT THE RIGHTHAND SIDE OF THE REGRESSION EQUATIONSPROVIDES A GOOD REPRESENTATION OF THE LEFTHAND SIDEBEGINFIGURET  CENTERLINEMBOXSUBFIGUREORIGINAL DATAEPSFIGFILEPICTUREDIRREGRESS1EPS                WIDTH045TEXTWIDTHQUADSUBFIGUREINTERPOLATED LINE AND                  ERRORSEPSFIGFILEPICTUREDIRREGRESS2EPS                WIDTH045TEXTWIDTH  CAPTIONDATA FOR REGRESSION  LABELFIGREGRESS1 TEST2REGRESSMENDFIGUREWE WILL CONSIDER IN DETAIL THE LINEAR REGRESSION PROBLEMREFEQREGRESS1  WE CAN STACK THE EQUATIONS TO OBTAINBEGINEQUATION BEGINBMATRIX Y1  Y2  VDOTS  YN ENDBMATRIX BEGINBMATRIX A X1  B  A X2  B  VDOTS  A XN  BENDBMATRIX  BEGINBMATRIX  E1  E2  VDOTS  ENENDBMATRIXLABELEQLINREGRESSENDEQUATIONFOR SOME ERROR TERMS EI  LET YBF  Y1 Y2 LDOTS YNTQQUAD EBF  E1E2 LDOTS ENT QQUADCBF  BEGINBMATRIX  A B ENDBMATRIXAND A  BEGINBMATRIX X1  1  X2  1  VDOTS  XN  1ENDBMATRIXTHEN REFEQLINREGRESS IS OF THE FORMBEGINEQUATIONYBF  A CBF  EBFLABELEQREGRESS4ENDEQUATIONWHICH AGAIN IS IN THE FORM REFEQMATLS SO THE BEST IN THELEASTSQUARES SENSE ESTIMATE OF CBF ISBEGINEQUATION CBF  AHA1AH YBFLABELEQ2REGRESSENDEQUATIONTHE LINE FOUND BY REFEQ2REGRESS MINIMIZES THE SUMS OF THESQUARES OF THE EM VERTICALDISTANCES BETWEEN THE DATA ABSCISSAS AND THE LINE AS SHOWN IN FIGUREREFFIGREGRESS1B  TO MINIMIZE EM SHORTEST DISTANCES OF THEDATA TO THE INTERPOLATING LINE THE METHOD OF EM TOTAL LEAST  SQUARES DISCUSSED IN SECTION REFSECTLS MUST BE USEDSINCE AHA IN REFEQ2REGRESS IS A MATSIZE22 MATRIXEXPLICIT CLOSEDFORM EXPRESSIONS FOR M AND B IN CBF CAN BEFOUND  THE SLOPE AND INTERCEPT FOR REAL DATA AREBEGINEQUATIONBEGINSPLITA  FRAC N SUMI1N XBARI YI  LEFTSUMI1N XIRIGHT  LEFTSUMJ1N  YIRIGHTN SUMI1N XI2     LEFTSUMI1N XIRIGHTLEFTSUMI1N XBARIRIGHT B  FRACLEFTSUMI1N XI2RIGHTLEFTSUMJ1N  YIRIGHT  LEFTSUMI1N XIRIGHTLEFTSUMI1N XBARI  YIRIGHT N SUMI1N XI2  LEFTSUMI1N XI  RIGHTLEFTSUMI1N XBARI RIGHTENDSPLITLABELEQLINREGRESSAENDEQUATIONBEGINEXAMPLE WEIGHTED LEASTSQUARES INDEXWEIGHTED LEASTSQUARES  FIVE MEASUREMENTS XIYII12LDOTS5 ARE MADE IN A SYSTEM  OF WHICH THE FIRST THREE ARE BELIEVED TO BE FAIRLY ACCURATE AND TWO  ARE KNOWN TO BE SOMEWHAT CORRUPTED BY MEASUREMENT NOISE  THE  MEASUREMENTS ARE 125  335  65  53  34FROM THESE FIVE MEASUREMENTS  THE DATA ARE TO BE FITTED TO A LINE ACCORDING TO THE MODEL Y  AX   B  THE MEASUREMENTS STACK UP IN THE MODEL EQUATION AS BEGINBMATRIX1  1  3  1  6  1  5  1  3  1 ENDBMATRIXBEGINBMATRIXA  B ENDBMATRIX  BEGINBMATRIX25  35  5  3  4ENDBMATRIX  EBFOR ACBF  YBF  EBFIN FINDING THE BEST MINIMUM SQUAREDERROR SOLUTION TO THIS PROBLEMIT IS APPROPRIATE TO WEIGHT MOST HEAVILY THOSE EQUATIONS WHICH AREBELIEVED TO BE THE MOST ACCURATE  LET W  DIAG10101011THEN USING REFEQWLS2 WE CAN DETERMINE THE OPTIMAL UNDER THEWEIGHTED INNER PRODUCT SET OF COEFFICIENTS  FIGUREREFFIGREGRESS2 ILLUSTRATES THE DATA AND THE LEASTSQUARES LINESFITTED TO THEM  THE ACCURATE DATA ARE PLOTTED WITH TIMES AND THEINACCURATE DATA ARE PLOTTED WITH CIRC  THE WEIGHTED LEASTSQUARESLINE FITS MORE CLOSELY ON AVERAGE TO THE MORE ACCURATE DATA WHILETHE UNWEIGHTED LEASTSQUARES LINE IS PULLED OFF SIGNIFICANTLY BY THEINACCURATE DATA AT X5BEGINFIGUREHTBP  CENTERLINE MBOXEPSFIGFILEPICTUREDIRREGRESS3EPS  CAPTIONILLUSTRATION OF LEASTSQUARES AND WEIGHTED LEASTSQUARES    APPROXIMATIONS  LABELFIGREGRESS2 TEST2REGRESS2MENDFIGUREENDEXAMPLEBEGINEXERCISES  ITEM GIVEN THE SET OF DATA X   225359 QQUAD Y  42 5 2 1243BEGINENUMERATEITEM MAKE A PLOT OF THE DATAITEM DETERMINE THE BEST LEASTSQUARES LINE THAT FITS THIS DATA AND  PLOT THE LINEITEM ASSUMING THAT THE FIRST AND LAST POINTS ARE BELIEVED TO BE THE  MOST ACCURATE FORMULATE A WEIGHTING MATRIX AND COMPUTE A WEIGHTED  LEASTSQUARES LINE THAT FITS THE DATA  PLOT THIS LINEENDENUMERATEITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS2 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS3 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION Y APPROX CEAX AS A LINEAR  REGRESSION PROBLEM WITH REGRESSION PARAMETERS C AND AITEM FORMULATE THE REGRESSION Y APPROX AXBT AS A LINEAR  REGRESSION PROBLEMITEM PERFORM THE COMPUTATIONS TO VERIFY THE SLOPE AND INTERCEPT OF  THE LINEAR REGRESSION IN REFEQLINREGRESSAITEM AS A MEASURE OF FIT IN A CORRELATION PROBLEM THE CORRELATION  COEFFICIENT ANALOGOUS TO REFEQCORRCOEFF CAN BE OBTAINED AS RHO  FRACLA XBF YBFRA  LA XBFONEBFRA LA YBFONEBFRA XBF  LA XBFONEBFRA YBF  LA YBFONEBFRATHE CORRELATION COEFFICIENT RHO  PM 1 IF X AND Y ARE EXACTLYFUNCTIONALLY RELATED AND RHO  0 IF THERE IS THEY ARE INDEPENDENTFOR THE LINEAR REGRESSION IN REFEQ2REGRESS DETERMINE ANEXPLICIT EXPRESSION FOR RHOITEM GIVEN A SET OF DATA XBFI I12LDOTS M WHERE EACH VECTOR  XBFI IN RBBD FORMULATE THE LEASTSQUARES SOLUTION TO FIND  THE BEST DDIMENSIONAL PLANE FITTING THIS DATAITEM LET US DEFINE AN INNER PRODUCT BETWEEN MATRICES X AND Y AS LA XY RA  TRACEXYHWHERE TRACECDOT IS THE SUM OF THE DIAGONAL ELEMENTS SEE SECTIONREFSECTPOSETRACE  WE WANT TO APPROXIMATE THE MATRIX Y BY THE SCALARLINEAR COMBINATION OF MATRICES X1X2LDOTSXM AS Y  C1 X1  C2 X2  CDOTS  CM XM  EUSING THE ORTHOGONALITY PRINCIPLE DETERMINE A SET OF NORMAL EQUATIONSTHAT CAN BE USED TO FIND C1C2LDOTSCM THAT MINIMIZE THEINDUCED NORM OF EITEM FOR THE ARMA INPUTOUTPUT RELATIONSHIP OF REFEQARMA  DETERMINE A SET OF LINEAR EQUATIONS FOR DETERMINING THE ARMA MODEL  PARAMETERS A1A2LDOTSAPB0B1LDOTSBQ ASSUMING  THAT THE MODEL OR PQ IS KNOWN AND THAT THE INPUT IS KNOWNENDEXERCISESSECTIONLEASTSQUARES FILTERINGLABELSECLSFILTIN THE LEASTSQUARES FILTER PROBLEM WE DESIRE TO FILTER A SEQUENCE OFINPUT DATA FT USING A FILTER WITH IMPULSE RESPONSE HT OFLENGTH M TO PRODUCE AN OUTPUT THAT MATCHES A DESIRED SEQUENCEDT AS CLOSELY AS POSSIBLE  EXAMPLES IN WHICH SUCH ACIRCUMSTANCE ARISES ARE GIVEN IN SECTION REFSECADFILT IN THECONTEXT OF ADAPTIVE FILTERING  IF WE CALL THE OUTPUT OF THE FILTERYT WE HAVE THE FILTER EXPRESSION YT   SUMI0M1 HI FTIWE CAN WRITE DT  YT  ET WHERE ET IS THE ERROR BETWEENTHE FILTER OUTPUT AND THE DESIRED FILTER OUTPUT DT  SUMI0M1 HI FTI  ETWE WANT TO CHOOSE THE FILTER COEFFICIENTS HI IN SUCH A WAYTHAT THE ERROR BETWEEN THE FILTER OUTPUT AND THE DESIRED SIGNAL SHOULDBE AS SMALL AS POSSIBLE THAT IS WE WANT TO MAKE ET  DT  YTSMALL FOR EACH TWHEN DOING EM LEASTSQUARES FILTERING INDEXLEASTSQUARES FILTERING THE CRITERION OF MINIMAL ERROR IS THATTHE SUM OF THE SQUARED ERRORS IS AS SMALL AS POSSIBLEBEGINEQUATION MIN SUMII1I2 EI2LABELEQLSNORMEENDEQUATIONWHERE I1 IS THE STARTING INDEX AND I2 THE ENDING INDEX OVERWHICH WE DESIRE TO MINIMIZE  THE SQUARED NORM IN REFEQLSNORMEIS INDUCED FROM THE INNER PRODUCT DEFINED BYBEGINEQUATION LA XBF YBF RA  SUMII1I2 XI YBARILABELEQLSIP01ENDEQUATIONLETTING YBF  BEGINBMATRIX YI1  YI11  VDOTS   YI2 ENDBMATRIX QQUAD HBF  BEGINBMATRIX H0  H1 VDOTS  HM1 ENDBMATRIX QQUAD XBF  BEGINBMATRIX XI1 XI11  VDOTS  XI2 ENDBMATRIXTHE INNER PRODUCT REFEQLSIP01 CAN BE WRITTEN AS LA XBFYBF RA  YBFH XBFAND THE FILTERED OUTPUTS CAN BE WRITTEN AS YBF  A HBFWHERE A IS A MATRIX OF THE INPUT DATA FT  THE MATRIX A TAKESVARIOUS FORMS DEPENDING ON THE ASSUMPTIONS MADE ON THE DATA ASDESCRIBED IN THE FOLLOWING  LET DBF  BEGINBMATRIX DI1  DI11  VDOTS  DI2ENDBMATRIXBE A VECTOR OF DESIRED OUTPUTS  THEN WE WANT DBF APPROX YBFWE CAN REPRESENT OUR APPROXIMATION PROBLEM AS DBF  A HBF  EBFWHERE EBF IS THE DIFFERENCE BETWEEN THE OUTPUT YBF AND THEDESIRED OUTPUT DBF  WE DESIRE TO FIND THE FILTER COEFFICIENTSHBF TO MINIMIZE  EBF   BY COMPARISON WITH REFEQMATLSOBSERVE THAT THE SOLUTION ISBEGINEQUATION HBF  AH A1 AH YBFLABELEQLSFILTSOLENDEQUATIONWE NOW EXAMINE THE FORM OF THE A MATRIX UNDER VARIOUS ASSUMPTIONSABOUT THE INPUTS  ASSUME THAT WE HAVE AVAILABLE TO US FOR THEPURPOSE OF FINDING THE COEFFICIENTS THE DATA F1F2LDOTSFNWITH A TOTAL OF N DATA POINTSBEGINDESCRIPTIONITEMTHE COVARIANCE METHOD INDEXCOVARIANCE  METHODCOVARIANCE METHOD IN THIS METHOD WE USE ONLY DATA THAT  IS EXPLICITLY AVAILABLE NOT MAKING ANY ASSUMPTIONS ABOUT DATA  OUTSIDE THIS SEGMENT OF OBSERVED DATA  THE DATA MATRIX A IN THIS  CASE IS THE MATSIZENM1M MATRIX A  BEGINBMATRIX FM  FM1  FM2  CDOTS  F1 FM1  FM  FM1  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 ENDBMATRIXLET QBFI BE THE MATSIZEM1 DATA VECTOR CORRESPONDING TO ACONJUGATED ROW OF A AS IN REFEQXSTACKROW THENBEGINEQUATION QBFI  BEGINBMATRIX FBARI   FBARI1   CDOTS   FBARIM1ENDBMATRIXLABELEQGRAMMXENDEQUATIONWITH THE NOTATION THAT FI  0 WHERE I IS OUTSIDE THE RANGE 1TO N AND WE CAN REPRESENT THE DATA MATRIX AS A  BEGINBMATRIX QBFMH  QBFM1H  VDOTS  QBFNHENDBMATRIXTHE GRAMMIAN CAN BE WRITTEN ASBEGINEQUATIONR  AH A  SUMIMN QBFI QBFHILABELEQGRAMM3ENDEQUATIONTHE GRAMMIAN R IS A HERMITIAN MATRIXITEMTHE AUTOCORRELATION METHOD INDEXAUTOCORRELATION    METHODAUTOCORRELATION METHOD IN THIS CASE WE ASSUME THAT  DATA PRIOR TO F1 AND AFTER FN ARE ALL ZERO  THE OUTPUT IS  TAKEN FROM I1  1 UP THROUGH I2  NM1 PRODUCING THE  MATSIZENM1M DATA MATRIX A  BEGINBMATRIX F1  0  0  CDOTS  0 F2  F1  0  CDOTS  0 F3  F2  F1  CDOTS  0 VDOTS FM  FM1  FM2  CDOTS  F1 FM1  FM  FM2  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 0  FN  FN1  CDOTS  FNM2 VDOTS 000 CDOTS  FN ENDBMATRIXTHE TERMS COVARIANCE METHOD AND AUTOCORRELATION METHOD DO NOTPRODUCE RESPECTIVELY A COVARIANCE MATRIX AND AN AUTOCORRELATIONMATRIX IN THE USUAL SENSE  RATHER THESE ARE THE TERMS FOR THESEMETHODS COMMONLY EMPLOYED IN THE SPEECH PROCESSING LITERATURE SEEEG CITEMAKHOUL1975  USING THE NOTATION OF REFEQGRAMMXWE CAN WRITE THE DATA MATRIX AS A  BEGINBMATRIX QBFH1 QBFT2  VDOTS  QBFTNM1 ENDBMATRIXIN A MANNER SIMILAR TO REFEQGRAMM3 WE CAN WRITE R  AH A  SUMI1NM1 QBFI QBFHITHIS IS A TOEPLITZ MATRIX INDEXTOEPLITZ MATRIXITEMPREWINDOWING METHOD INDEXPREWINDOWING METHOD IN THIS METHOD WE ASSUME THAT FT0 FOR  T 1 AND USE DATA UP TO FN SO THAT I1  1 AND I2  N  THEN THE DATA MATRIX IS THE MATSIZENM MATRIXBEGINEQUATION A  BEGINBMATRIXF1  0  0  CDOTS  0 F2  F1  0  CDOTS  0 F3  F2  F1  CDOTS  0 VDOTS FM  FM1  FM2  CDOTS  F1 FM1  FM  FM2  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 ENDBMATRIX  BEGINBMATRIX QBFH1  QBFH2  VDOTS   QBFHN ENDBMATRIXLABELEQPREWINDOW1ENDEQUATIONAND R  SUMI1N QBFIQBFHIITEMPOSTWINDOWING METHOD WE BEGIN WITH I1M AND ASSUME THAT  DATA AFTER N ARE EQUAL TO ZERO  THEN A IS THE MATSIZENM MATRIX A  BEGINBMATRIXFM  FM1  FM2  CDOTS  F1 FM1  FM  FM2  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 0  FN  FN1  CDOTS  FNM2 VDOTS 000 CDOTS  FN ENDBMATRIXAND R  SUMIMMN QBFIQBFHIENDDESCRIPTIONBEGINEXAMPLE  SUPPOSE WE OBSERVE THE DATA SEQUENCE  F1 LDOTS F5   12345WHICH WE WANT TO FILTER WITH A FILTER OF LENGTH M3  THE DATAMATRICES CORRESPONDING TO EACH INTERPRETATION LABELED RESPECTIVELYATEXT COV ATEXT AC ATEXT PRE AND ATEXT  POST WITH THEIR CORRESPONDING GRAMMIANS ARE SHOWN HEREBEGINALIGNED ATEXT COV  BEGINBMATRIXHFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 ENDBMATRIXQQUADATEXT AC  BEGINBMATRIXHFILL 1  HFILL 0  HFILL 0 HFILL 2  HFILL 1  HFILL 0 HFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 HFILL 0  HFILL 5  HFILL 4 HFILL 0  HFILL 0  HFILL 5 ENDBMATRIX  EQNSKIPATEXTPRE  BEGINBMATRIXHFILL 1  HFILL 0  HFILL 0 HFILL 2  HFILL 1  HFILL 0 HFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 ENDBMATRIX QQUADATEXTPOST  BEGINBMATRIXHFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 HFILL 0  HFILL 5  HFILL 4 HFILL 0  HFILL 0  HFILL 5 ENDBMATRIXENDALIGNEDBEGINALIGNEDRTEXT COV  BEGINBMATRIXHFILL 50  HFILL 38  HFILL 26 HFILL 38  HFILL 29  HFILL 20 HFILL 26  HFILL 20  HFILL 14 ENDBMATRIX QQUADRTEXT AC  BEGINBMATRIXHFILL 55  HFILL 40  HFILL 26 HFILL 40  HFILL 55  HFILL 40 HFILL 26  HFILL 40  HFILL 55 ENDBMATRIX  RTEXT PRE  BEGINBMATRIXHFILL 55  HFILL 40  HFILL 26 HFILL 40  HFILL 30  HFILL 20 HFILL 26  HFILL 20  HFILL 14 ENDBMATRIX QQUADRTEXT POST  BEGINBMATRIXHFILL 50  HFILL 38  HFILL 26 HFILL 38  HFILL 54  HFILL 40 HFILL 26  HFILL 40  HFILL 55 ENDBMATRIXENDALIGNEDENDEXAMPLEOBSERVE THAT WHILE ALL OF THE DATA MATRICES ARE TOEPLITZ CONSTANTALONG THE DIAGONALS THE ONLY GRAMMIAN WHICH IS TOEPLITZ IS THEONE WHICH ARISES FROM THE AUTOCOVARIANCE FORM OF THE DATA MATRIXSC MATLAB CODE TO COMPUTE THE LEASTSQUARES FILTER COEFFICIENTS ISGIVEN IN ALGORITHM REFALGLSFILT  BEGINNEWPROGENVLEASTSQUARES FILTER COMPUTATIONLSFILTM   LSFILTLEASTSQUARES FILTER ENDNEWPROGENV BEGINEXAMPLE   FOR THE INPUT DATA OF THE PREVIOUS EXAMPLE THE FOLLOWING DESIRED   DATA ARE KNOWN DBF  2 5 1117 2317 15TWE WANT TO FIND A FILTER OF LENGTH M3 THAT PRODUCES THIS DATAUSING THE FOUR DIFFERENT DATA SETS IN THE EXAMPLE WITH SELECTIONS OFDBF CORRESPONDING TO THE DATA USED WE OBTAIN FROM THE SC MATLABCOMMANDS SMALLSKIPBEGINALLTTINDENT HCV  LSFILTFD3531INDENT HAC  LSFILTFD32INDENT HPRE  LSFILTFD1533INDENT HPOST  LSTILFFD3734SMALLSKIPENDALLTTTHE FILTER COEFFICIENTS HBFTEXTCOV  BEGINBMATRIX15225 ENDBMATRIXT QQUADHBFTEXTAUTO  BEGINBMATRIX 213 ENDBMATRIXT HBFTEXTPRE  BEGINBMATRIX 213 ENDBMATRIXT QQUADHBFTEXTPOST  BEGINBMATRIX 213 ENDBMATRIXTRESPECTIVELYENDEXAMPLEBEGINEXAMPLE  AN APPLICATION OF LEASTSQUARES FILTERING IS ILLUSTRATED IN FIGURE  REFFIGLSEQ IN A CHANNEL EQUALIZER  INDEXEQUALIZERLEASTSQUARES APPLICATION  A SEQUENCE OF BITS  BT IS PASSED THROUGH A DISCRETETIME CHANNEL WITH UNKNOWN  IMPULSE RESPONSE THE OUTPUT OF WHICH IS CORRUPTED BY NOISE  TO  COUNTERACT THE EFFECT OF THE CHANNEL THE SIGNAL IS PASSED THROUGH  AN EQUALIZER WHICH IN THIS CASE IS AN FIR FILTER WHOSE COEFFICIENTS  HAVE BEEN DETERMINED USING A LEASTSQUARES CRITERION  IN ORDER TO  DETERMINE WHAT THE COEFFICIENTS ARE SOME SET OF KNOWN DATA  A  EM TRAINING SEQUENCE  IS USED AT THE BEGINNING OF THE  TRANSMISSION  THIS SEQUENCE IS DELAYED AND USED AS THE DESIRED  SIGNAL DT  USING THIS TRAINING SEQUENCE THE FILTER  COEFFICIENTS HK ARE COMPUTED BY USING REFEQLSFILTSOL  AFTER WHICH THE COEFFICIENTS ARE LOADED INTO THE EQUALIZER FILTER    THIS EXAMPLE IS MORE A DEMONSTRATION OF A CONCEPT THAN A PRACTICAL  REALITY  WHILE EQUALIZERS ARE COMMON ON MODERN MODEM TECHNOLOGY  THEY ARE MORE COMMONLY IMPLEMENTED USING ADAPTIVE FILTERS  ADAPTIVE  EQUALIZERS ARE EXAMINED IN SECTION REFSECRLSEX RLS ADAPTIVE  EQUALIZER AND SECTION REFSECLMS LMS ADAPTIVE EQUALIZERENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREQUALIZER1    CAPTIONLEASTSQUARES EQUALIZER EXAMPLE    LABELFIGLSEQ  ENDCENTERENDFIGUREBEGINEXERCISES  ITEM VERIFY REFEQGRAMM3ITEM FOR THE DATA SEQUENCE 11235813  BEGINENUMERATE  ITEM WRITE DOWN THE DATA MATRIX A AND THE GRAMMIAN AH A USING    I THE COVARIANCE AND II THE AUTOCORRELATION METHODS  ITEM WE DESIRE TO USE THIS SEQUENCE TO TRAIN A SIMPLE LINEAR    PREDICTOR  THE DESIRED SIGNAL DT IS THE VALUE OF XT    AND THE DATA USED ARE THE TWO PRIOR SAMPLES  THAT IS  XT  A1 XT1  A2 XT2  ET WHERE ET THE PREDICTION ERRORDETERMINE THE LEASTSQUARES COEFFICIENTS FOR THE PREDICTOR USING THECOVARIANCE AND AUTOCORRELATION METHODSITEM DETERMINE THE MINIMUM LEASTSQUARES ERROR FOR BOTH METHODS  ENDENUMERATEITEM BF COMPUTER EXERCISE GENERATE A SEQUENCE OF N RANDOM PM  1 BITS AND PASS THEM THROUGH A CHANNEL WITH TRANSFER FUNCTION HZ  FRAC119Z1THEN ADD NOISE WITH VARIANCE SIGMAN2  01  DETERMINE ALEASTSQUARES FILTER FOR THIS CHANNEL FOR VARIOUS VALUES OF THEDELAYENDEXERCISESSUBSECTIONLEASTSQUARES PREDICTION AND AR SPECTRUM ESTIMATIONLABELSECLSSPECTESTINDEXSPECTRUM ESTIMATION CONSIDER NOW THE ESTIMATION PROBLEM INWHICH WE DESIRE TO PREDICT XT USING A LINEAR PREDICTORINDEXLINEAR PREDICTOR BASED UPON XT1 XT2 LDOTS XTMWE THEN HAVEBEGINEQUATION XT  SUMI1M AI XTI  FTLABELEQLSARSEENDEQUATIONUSING AI  HI AS THE COEFFICIENTS WHERE FT IS NOW USED TODENOTE THE FORWARD PREDICTOR INDEXLINEAR PREDICTORFORWARD ERRORTHE PREDICTOR OF REFEQLSARSE IS CALLED A EM FORWARD  PREDICTOR  THIS IS ESSENTIALLY THE PROBLEM SOLVED IN THE LASTSECTION IN WHICH THE DESIRED SIGNAL IS THE SAMPLE DT  XT ANDTHE DATA USED ARE THE EM PREVIOUS DATA SAMPLES  WE CAN MODEL THESIGNAL XT AS BEING THE OUTPUT OF A SIGNAL WITH INPUT FT WHERETHE SYSTEM FUNCTION IS HZ  FRACXZFZ  FRAC11  SUMI1N AI ZI FRAC1AZIF FT IS A RANDOM SIGNAL WITH POWER SPECTRAL DENSITY PSD SFZTHEN THE PSD OF XT ISBEGINEQUATION SXZ  FRAC11SUMI1M AI ZI1 SUMI1M  ABARI ZIPFZ   FRAC1AZABAR1ZLABELEQFORWARDPSDENDEQUATIONIF FT IS ASSUMED TO BE A WHITENOISE SEQUENCE WITHVARIANCE SIGMAF2 THEN THE RANDOM PROCESS XT HAS THE PSD SXZ  FRACSIGMAF2AZABAR1ZEVALUATING THIS ON THE UNIT CIRCLE ZEJOMEGA WE OBTAINBEGINEQUATION SXOMEGA DEFEQ BIGL SXZBIGRZEJOMEGA    FRACSIGMAF21    SUMI1M AI EJOMEGA     I2  FRACSIGMAF2AOMEGA2LABELEQPSD2ENDEQUATIONTHUS BY FINDING THE COEFFICIENTS OF THE LINEAR PREDICTOR WE CAN DETERMINEAN ESTIMATE OF THE SPECTRUM UNDER THE ASSUMPTION THAT THE SIGNAL ISPRODUCED BY THE AR MODEL REFEQLSARSE  WE CAN OBTAIN MORE DATA TO PUT IN OUR DATA MATRIX AND USUALLYDECREASE THE VARIANCE OF THE ESTIMATE BY USING A EM BACKWARD  PREDICTOR IN ADDITION TO A FORWARD PREDICTOR  INDEXLINEAR  PREDICTORBACKWARD  IN THE BACKWARD PREDICTOR THE M DATA POINTS XTXT1 LDOTS XTM1 ARE USED TO ESTIMATE XTM XTM  SUMI1M AI XTMI  BTWHERE BT IS THE BACKWARD PREDICTION ERROR  AS BEFORE IF WE VIEWXTM AS THE OUTPUT OF A SYSTEM DRIVEN BY AN INPUT BT WEOBTAIN A SYSTEM FUNCTION HBZ  FRACXZBZ  FRAC1ZM1  SUMI1M  ABARI ZI  FRAC1ZMABAR1ZIF BT IS A WHITENOISE SEQUENCE WITH VARIANCE SIGMAB2 SIGMAF2 THEN THE PSD OF THE SIGNAL XTM ISBEGINEQUATIONSXZ  SIGMAB2 FRAC1ABAR1ZAZLABELEQBACKWARDPSDENDEQUATIONTHE SAME AS IN REFEQFORWARDPSD  SINCE BOTH THE FORWARDPREDICTOR AND THE BACKWARD PREDICTOR USE THE SAME PREDICTORCOEFFICIENTS JUST CONJUGATED AND IN A DIFFERENT ORDER WE CAN USETHE BACKWARD PREDICTOR INFORMATION TO IMPROVE OUR ESTIMATE OF THECOEFFICIENTS  IF WE HAVE MEASURED DATA X1 X2 LDOTS XN WEWRITE OUR PREDICTION EQUATIONS AS FOLLOWS USING THE COVARIANCE METHODEMPLOYING ONLY MEASURED DATA BEGINBMATRIX XM  XM1  CDOTS  X1 XM1  XM  CDOTS  X2 VDOTS XN1  XN2  CDOTS  XNM XBF2  XBF3  CDOTS  XBFM1 XBAR3  XBAR4  CDOTS  XBARM2 VDOTS XBARNM1  XBARNM2  CDOTS  XBARN ENDBMATRIXBEGINBMATRIX A1  A2  VDOTS  AM ENDBMATRIXBEGINBMATRIX XM1   XM2   VDOTS XN  XBAR1  XBAR2  VDOTS XBARNMENDBMATRIX BEGINBMATRIX FM1  FM2  VDOTS  FN  BBARNM1  BBARNM2  VDOTS  BBARNM ENDBMATRIXLET US WRITE THIS AS XBF  A HBF  EBFWHERE XBF AND EBF NOW ARE MATSIZE2NM1 AND A ISMATSIZE2NMN  IN THE DATA MATRIX THE FIRST NM ROWSCORRESPOND TO THE FORWARD PREDICTOR AND THE SECOND NM ROWSCORRESPOND TO THE BACKWARD PREDICTOR  OUR OPTIMIZATION CRITERION ISTO MINIMIZE SUMIN1N FI2  BI2AS BEFORE A LEASTSQUARES SOLUTION IS STRAIGHTFORWARD  THISTECHNIQUE OF SPECTRUM ESTIMATION IS KNOWN AS THE FORWARDBACKWARDLINEAR PREDICTION FBLP INDEXLINEAR PREDICTORFORWARDBACKWARDTECHNIQUE OR THE MODIFIED COVARIANCE TECHNIQUE  INDEXMODIFIED  COVARIANCE METHOD AN ESTIMATE OF THE VARIANCE IS SIGMAHATF2  SIGMAHATB2   EBFMIN22A SC MATLAB FUNCTION THAT COMPUTES THE AR PARAMETERS USING THEMODIFIED COVARIANCE TECHNIQUE IS SHOWN IN ALGORITHM REFALGFBLPBEGINNEWPROGENVFORWARDBACKWARD LINEAR PREDICTOR    ESTIMATEFBLPMFBLPFORWARDBACKWARD LINEAR PREDICTORENDNEWPROGENVSECTIONMINIMUM MEANSQUARE ESTIMATIONLABELSECMMSINDEXMINIMUM MEANSQUAREIN THE LEASTSQUARES ESTIMATION OF THE PRECEDING SECTIONS WE HAVE NOTEMPLOYED NOR ASSUMED THE EXISTENCE OF ANY PROBABILISTIC MODEL  THEOPTIMIZATION CRITERION HAS BEEN TO MINIMIZE THE SUM OF SQUARED ERRORIN THIS SECTION WE CHANGE OUR VIEWPOINT SOMEWHAT BY INTRODUCING APROBABILISTIC MODEL FOR THE DATALET P1 P2 LDOTS PM BE RANDOM VARIABLES  WE DESIRE TO FINDCOEFFICIENTS CI TO ESTIMATE THE RANDOM VARIABLE X USING X  C1 P1  C2 P2  CDOTS  CM PM  EIN SUCH A WAY THAT THE NORM OF THE SQUARED ERROR IS MINIMIZEDUSING THE INNER PRODUCTBEGINEQUATIONLA X Y RA  EX YBARLABELEQMMSE0ENDEQUATIONTHE MINIMUM MEANSQUARE ESTIMATE OF CBF IS GIVEN BY  RCBF  PBFWHEREBEGINEQUATION R  BEGINBMATRIXEP1PBAR1  EP2PBAR1  CDOTS  EPM PBAR1 EP1PBAR2  EP2PBAR2  CDOTS  EPM PBAR2 VDOTS EP1PBARM  EP2PBARM  CDOTS  EPM PBARM  ENDBMATRIXQUAD TEXTANDQUADPBF  BEGINBMATRIXEXPBAR1  EXPBAR2  VDOTS  EXPBARM ENDBMATRIXLABELEQMMSERDENDEQUATIONTHE MINIMUM MEANSQUARED ERROR IN THIS CASE IS GIVEN USINGREFEQEMIN3 ASBEGINEQUATIONBEGINSPLITEMIN  SIGMAX2  PBFH R1 PBF  SIGMAX2  PBFH CBFENDSPLITLABELEQEMINMMSENDEQUATIONBEGINEXAMPLE LABELEXMMMSEPRED  SUPPOSE THAT  ZBF  X1X2X3TIS A REAL GAUSSIAN RANDOM VECTOR WITH MEAN ZERO AND COVARIANCE RZZ  COVZBF  EZBFZBFT  BEGINBMATRIX 1  2  1 2  2  3 1  3  4 ENDBMATRIXGIVEN MEASUREMENTS OF X1 AND X2 WE WISH TO ESTIMATE X3 USINGA LINEAR ESTIMATOR XHAT3  C1 X1  C2 X2THE NECESSARY CORRELATION VALUES IN REFEQMMSERD CAN BE OBTAINEDFROM THE COVARIANCE RZZ R  BEGINBMATRIX EX1X1  EX1X2  EX2X1  EX2  X2 ENDBMATRIX  BEGINBMATRIX 1  2  2  2 ENDBMATRIX QQUADTEXTAND QQUAD PBF  BEGINBMATRIXEX3 X1  EX3 X2ENDBMATRIX BEGINBMATRIX 1  3ENDBMATRIXFROM WHICH THE OPTIMAL COEFFICIENTS ARE CBF  BEGINBMATRIX 00714  01429 ENDBMATRIXTHE MINIMUM MEANSQUARED ERROR IS  EMIN  4  PBFT R1PBF  395ENDEXAMPLESECTIONMINIMUM MEANSQUARED ERROR MMSE FILTERINGLABELSECMMSSEFILTINDEXMINIMUM MEANSQUAREFILTERING A MINIMUM MEANSQUARE MMSFILTER INDEXWIENER FILTER IS MATHEMATICALLY SIMILAR TO TO ALEASTSQUARES FILTER EXCEPT THAT THE EXPECTATION OPERATOR IS USED ASTHE INNER PRODUCT  GIVEN A SEQUENCE OF DATA  FT  WE DESIRETO DESIGN A FILTER IN SUCH A WAY THAT WE GET AS CLOSE AS POSSIBLE TOSOME DESIRED SEQUENCE DT  IN THE INTEREST OF GENERALITY WEASSUME THE POSSIBILITY OF AN IIR FILTERBEGINEQUATIONYT   SUML0INFTY HL FTLLABELEQMMSE1ENDEQUATIONIN ADOPTING A STATISTICAL MODEL WE ASSUME THAT THE SIGNALS INVOLVEDARE WIDESENSE STATIONARY SO THAT FOR EXAMPLE EXT  EXTLQQUAD TEXTFOR ALL  LAND EXTXBARTLDEPENDS ONLY UPON THE TIME DIFFERENCE L AND NOT UPON THE SAMPLEINSTANT TUSINGBEGINEQUATION ET  DT  YTLABELEQMMSE2ENDEQUATIONAS THE ESTIMATOR ERROR BY THE ORTHOGONALITY PRINCIPLE THE SQUAREDNORM OF ERROR WHICH IN THIS CASE IS TERMED THE EM MEANSQUARED  ERROR  ET2  EET EBARTIS MINIMIZED WHEN THE ERROR IS ORTHOGONAL TO THE DATA  THAT IS THEOPTIMAL ESTIMATOR SATISFIESLA DT  SUML0INFTY HLFTLFTI RA  0FOR I012LDOTS ORBEGINEQUATION  LABELEQMMSE3  LA DTFTI RA  SUML0INFTY HL LA  FTLFTIRA  0 ENDEQUATIONUSING THE INNER PRODUCT REFEQMMSE0 WE OBTAINBEGINEQUATION  LABELEQMMSE4  SUML0INFTY HL EFTLFBARTI  EFBARTIDTENDEQUATIONEQUATION REFEQMMSE4 IS AN INFINITE SET OF NORMAL EQUATIONSFOR THIS CASE IN WHICH THE INNER PRODUCT ISDEFINED USING THE EXPECTATION THE NORMAL EQUATIONS ARE REFERRED TO ASTHE EM WIENERHOPF EQUATIONS   INDEXWIENERHOPF EQUATIONSWE CAN PLACE THE NORMAL EQUATIONS INTO A MORE STANDARD FORM BYEXPRESSING THE GRAMMIAN IN THE FORM OF AN AUTOCORRELATION MATRIXDEFINE RIL  EFTLFBARTI  LA FTLFTIRAAND PI  EFBARTIDT  LA DTFTIRAAND OBSERVE THAT RK  RBARK  THEN REFEQMMSE4 CAN BEWRITTEN ASBEGINEQUATIONSUML0INFTY HL RIL  PIQQUAD I01LDOTSLABELEQMMSE4AENDEQUATIONSOLUTION OF THIS PROBLEM FOR AN IIR FILTER IS REEXAMINED IN SECTIONREFSECFREQFILT  FOR NOW WE FOCUS ON THE SOLUTION WHEN H IS AN FIR FILTER WITHM COEFFICIENTS  THEN THE FILTER OUTPUT CAN BE WRITTEN AS YT  FBFTH HBFWHEREBEGINEQUATION FBFT  BEGINBMATRIXFBART FBART1 LDOTS FBARTM1ENDBMATRIXTLABELEQDEFFENDEQUATIONNOTE THE CONJUGATES IN THIS DEFINITION AND HBF  BEGINBMATRIX H0  H1 LDOTS  HM1ENDBMATRIXTUNDER THE ASSUMPTION OF AN FIR FILTER REFEQMMSE4A CAN BE WRITTENASBEGINEQUATIONSUML0M1 HL RIL  PIQQUAD I01LDOTSLABELEQMMSE5ENDEQUATIONWHICH WE CAN EXPRESS IN MATRIX FORM WITH RIL  RILBEGINEQUATION  LABELEQMMSE6  R HBF  PBFENDEQUATIONWHEREBEGINEQUATIONBEGINSPLITR  BEGINBMATRIX R0  RBAR1  RBAR2 CDOTS  RBARM1 R1  R0  RBAR1  CDOTS  RBARM2 R2  R1  R0  CDOTS  RBARM3 VDOTS RM1  RM2  RM3  CDOTS  R0 ENDBMATRIX  EFBFTFBFHTENDSPLITLABELEQREFFENDEQUATIONANDBEGINEQUATIONLABELEQPEFDBEGINSPLITPBF  BEGINBMATRIX P0   P1  P2  CDOTS PM1ENDBMATRIX  EFBFTDTENDSPLITENDEQUATIONTHE OPTIMAL WEIGHTS FROM REFEQMMSE6 ARE HBF  R1 PBFTHE MATRIX R IS THE GRAMMIAN MATRIX AND HAS THE SPECIAL FORM OF ATOEPLITZ MATRIX INDEXTOEPLITZ MATRIX BEING CONSTANT ON THEDIAGONALS  BECAUSE OF THIS SPECIAL FORM FAST ALGORITHMS EXIST FORINVERTING THE MATRIX AND SOLVING FOR THE OPTIMUM FILTER COEFFICIENTSTOEPLITZ MATRICES ARE DISCUSSED FURTHER IN SECTION REFSECTOEPLITZWE HAVE ALREADY SEEN ONE EXAMPLE OF THE SOLUTION OF TOEPLITZEQUATIONS WITH A SPECIAL RIGHTHAND SIDE IN MASSEYS ALGORITHM INSECTION REFSECLFSR1THE MINIMUM MEANSQUARED ERROR CAN BE DETERMINED USING REFEQEMINMMSTO BE   E2MIN  EE2MIN   D2   Y2USING THE NOTATION E2  SIGMAE2 AND D2  SIGMAD2AND NOTING THATBEGINALIGNED YT2  EYT YBART  EHBFHT XBFT  XBFHT HBF  HBFH R HBF  PBFH HBFENDALIGNEDWE OBTAIN BEGINEQUATION SIGMAE2MIN  SIGMAD2  PBFH HBFLABELEQEMINWIENENDEQUATIONBEGINEXAMPLE LABELEXMEQ1INDEXEQUALIZERMINIMUM MEANSQUARE  IN THIS EXAMPLE WE EXPLORE A SIMPLE EQUALIZER  SUPPOSE WE HAVE A  CHANNEL WITH TRANSFER FUNCTION  HCZ  FRAC116 Z1PASSING INTO THE CHANNEL IS A DESIRED SIGNAL DT  THE OUTPUT OFTHE CHANNEL IS UT SO THAT WE HAVEBEGINEQUATION  LABELEQWFEX1  UT  06 UT1  DTENDEQUATION  HOWEVER WE OBSERVE ONLY A NOISECORRUPTEDVERSION OF THE CHANNEL OUTPUT FT  UT  NTWHERE NT IS A ZEROMEAN WHITENOISE SEQUENCE WITH VARIANCESIGMAN2  016 WHICH IS UNCORRELATED WITH NUT  SUPPOSEFURTHERMORE THAT WE HAVE A STATISTICAL MODEL FOR THE DESIRED SIGNALIN WHICH WE KNOW THAT DT IS A FIRSTORDER AR SIGNAL GENERATED BY DT  5 DT1  NUTWHERE NUT IS A ZEROMEAN WHITENOISE SIGNAL WITH VARIANCESIGMANU2  01  BASED ON THIS INFORMATION WE DESIRE TO FIND ANOPTIMAL WIENER FILTER TO ESTIMATE DT USING THE OBSERVED SEQUENCEFT  THE DIAGRAM IS SHOWN IN FIGURE REFFIGWFEX1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREQUALIZER2    CAPTIONAN EQUALIZER PROBLEM    LABELFIGWFEX1  ENDCENTERENDFIGURETHE CASCADE OF THE AR PROCESS AND THE CHANNEL GIVES THE COMBINEDTRANSFER FUNCTION FROM NUT TO UT AS HZ  FRAC115Z116Z1  FRAC11  1Z1   3 Z2  FRAC11  A1 Z1  A2 Z2SO THAT UT 1 UT1 3 UT2  NUTIN THIS EXAMPLE SINCE THE CHANNEL OUTPUT IS AN AR2 PROCESS THEEQUALIZER USED IS A TWOTAP FIR FILTER WE NEED THE MATRIX R CONTAINING AUTOCORRELATIONS OF THE SIGNALFT AND THE CROSSCORRELATION VECTOR PBF  SINCE FT  UT NT AND SINCE NUT AND NT ARE UNCORRELATED WE HAVE R  RFF  RUU  RNNWHERE RUU IS THE AUTOCORRELATION MATRIX FOR THE SIGNAL UT ANDRNN IS THE AUTOCORRELATION MATRIX FOR THE SIGNAL NT  SINCENT IS A WHITENOISE SEQUENCE RNN  SIGMAN2 I WHERE IIS THE MATSIZE22 IDENTITY MATRIX  TO FIND RUU  BEGINBMATRIX RU0  RU1  RU1  RU0 ENDBMATRIXWE USE THE RESULTS FROM  SECTION REFSECARPROCESS  SPECIFICALLYFROM REFEQYW7 AND REFEQYW6 WE FIND BEGINALIGNEDSIGMAU2  RU0 LEFTFRAC1A21A2RIGHTFRACSIGMANU21A22    A12  01122 RU1  FRACA11A2SIGMAU2  00160ENDALIGNEDTHUS R  BEGINBMATRIX 16  0  0  16 ENDBMATRIX BEGINBMATRIX1122  0160  0160  1122  ENDBMATRIX  BEGINBMATRIX 2722   0160  0160  HFILL 2722 ENDBMATRIXFOR THE CROSSCORRELATION VECTORBEGINALIGNEDPBF  EBEGINBMATRIX FBARTDT  FBART1DTENDBMATRIX  EBEGINBMATRIXUBARTNBARTDT UBART1NBART1DT ENDBMATRIX  EBEGINBMATRIX UBARTDT  UBART1DT ENDBMATRIXENDALIGNEDSINCE DT IS UNCORRELATED WITH NTN  MULTIPLYINGREFEQWFEX1 THROUGH BY UBARTK AND TAKING EXPECTATIONS WEOBTAIN PK  EUBARTKDT  RUK  06 RUK1FROM WHICH WE CAN DETERMINE PBF BEGINBMATRIX HFILL 01206  HFILL 00513ENDBMATRIXTHE OPTIMAL FILTER COEFFICIENTS ARE HBF  R1 PBF  BEGINBMATRIXHFILL 03893 HFILL 02113 ENDBMATRIXTO COMPUTE THE MINIMUM MEANSQUARED ERROR FROM REFEQEMINWIEN WENEED SIGMAD2  THIS IS FOUND USING REFEQFIRSTARVAR AS SIGMAD2  FRACSIGMANU2152THEN SIGMAE2  00826THE ERROR SURFACE IS OBTAINED BY PLOTTING SEE REFEQGRADMIN2 JHBF  SIGMAD2  2 PBFTBEGINBMATRIXH0 H1ENDBMATRIX  H0 H1R BEGINBMATRIX H0  H1ENDBMATRIXAS A FUNCTION OF H0H1  FIGURE REFFIGWFTESTCONTSHOWS A CONTOUR PLOT OF THE ERROR SURFACEBEGINFIGURET USE WFTESTCONT AFTER RUNNING WFTESTCENTERINGEPSFIGFILEPICTUREDIRWFTESTCONTEPS  CAPTIONCONTOUR PLOT OF AN ERROR SURFACE  LABELFIGWFTESTCONTENDFIGURE WFTESTM WFTESTCONTMALGORITHM REFALGWIENFILT1 IS SC MATLAB CODE DEMONSTRATING THESECOMPUTATIONSBEGINNEWPROGENVTWOTAP CHANNEL    EQUALIZERWFTESTMWIENFILT1TWOTAP CHANNEL EQUALIZERENDNEWPROGENVENDEXAMPLEANOTHER EXAMPLE OF MMSE FILTER DESIGN IS GIVEN IN CONJUNCTION WITH THERLS FILTER IN REFSECRLSEXBEGINEXERCISESITEM IMPLEMENTATION OF EQUALIZER IN EXAMPLE REFEXMEQ1 GENERATE  THE SIGNAL FT IN EXAMPLE REFEXMEQ1 BY CREATING AN AR1  SIGNAL WHICH IS PASSED THROUGH HCZ THEN CORRUPTED BY ADDITIVE  NOISE  THEN PASS THIS SIGNAL THROUGH THE EQUALIZER DESIGNED IN THAT  EXAMPLE  COMPARE THE EQUALIZED SIGNAL WITH THE SIGNAL DT  ITEM FOR A DATA SEQUENCE  XT THE CORRELATION MATRIX R IS R  BEGINBMATRIX 5  3  3  5 ENDBMATRIXAND THE CROSS CORRELATION VECTOR PBF WITH A DESIRED SIGNAL IS PBF  BEGINBMATRIX 2  5 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL WEIGHT VECTORITEM DETERMINE THE MINIMUM MEANSQUARED ERRORENDENUMERATEITEM FOR A ZEROMEAN RANDOM VECTOR XBF  X1X2X3 WITH  COVARIANCE COVXBF  EXBFXBFT  BEGINBMATRIX 1  7  5  7  4  2 5  2  3 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL COEFFICIENTS OF THE PREDICTOR OF X1 IN TERMSOF X2 AND X3 XHAT1  C1 X2  C2 X3ITEM DETERMINE THE MINIMUM MEANSQUARED ERRORITEM HOW IS THIS ESTIMATOR MODIFIED IF THE MEAN OF XBF IS E XBF   123TENDENUMERATEITEM CITEHAYKIN1996 A RADAR SIGNAL IS TRANSMITTED AS  ST  A0 EJOMEGA0 TTHE SAMPLED RECEIVED SIGNALS ARE REPRESENTED AS XT  A1 EJOMEGA1 T  NUTWHERE OMEGA1 IS THE RECEIVED SIGNAL FREQUENCY IN GENERALDIFFERENT FROM OMEGA0 BECAUSE OF DOPPLER SHIFT AND NUT IS AWHITE NOISE SIGNAL WITH VARIANCE SIGMAN2  LET XBFT  X0X1LDOTS XM1TBE A VECTOR OF RECEIVED SIGNAL SAMPLESBEGINENUMERATEITEM SHOW THAT R  EXBFT XBFHT  SIGMANU2 I  SIGMA1 SBFOMEGA1SBFHOMEGA1WHERE SBFOMEGA1  1EJOMEGA1EJ2OMEGA1LDOTSEM1  J OMEGA1TITEM THE TIME SERIES XT IS APPLIED TO AN FIR WIENER FILTER WITH  M COEFFICIENTS IN WHICH THE CROSSCORRELATION BETWEEN XT AND  THE DESIRED SIGNAL DT IS PRESET TO PBF  SBFOMEGA0DETERMINE AN EXPRESSION FOR THE TAPWEIGHT VECTOR OF THE WIENER FILTERENDENUMERATEITEM A CHANNEL WITH TRANSFER FUNCTION H2Z  FRAC112Z1AND OUTPUT UT IS DRIVEN BY AN AR1 SIGNAL DT GENERATED BY DT  4 DT1  NUTWHERE NUT IS A ZEROMEAN WHITE NOISE SIGNAL WITH SIGMANU2 2  THE CHANNEL OUTPUT IS CORRUPTED BY NOISE NT WITH VARIANCESIGMAN2  1 TO PRODUCE THE SIGNAL FT  UT  NTDESIGN A SECONDORDER WIENER EQUALIZER TO MINIMIZE THE AVERAGE SQUAREDERROR BETWEEN FT AND DTITEM LABELEXLINPRED BF LINEAR PREDICTION A COMMON APPLICATION  OF WIENER FILTERING IS IN THE CONTEXT OF LINEAR PREDICTION  LET  DT  XT BE THE DESIRED VALUE AND LET XHATT  SUMI1M WFI XTIBE THE PREDICTED VALUE OF XT USING AN MTH ORDER PREDICTOR BASEDUPON THE MEASUREMENTS XT1XT2LDOTSXTM ANDLET FMT  XT  XHATTBE THE EM FORWARD PREDICTION ERROR  INDEXFORWARD  PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORFORWARD THEN FMT  SUMI0M AFI XTIWHERE AF0  1 AND AFI  WFI I12LDOTSMASSUME THAT XT IS A ZEROMEAN RANDOM SEQUENCE  WE DESIRE TODETERMINE THE OPTIMAL SET OF COEFFICIENTS WFII12LDOTSMTO MINIMIZE EFMT2BEGINENUMERATEITEM USING THE ORTHOGONALITY PRINCIPLE WRITE DOWN THE NORMAL  EQUATIONS CORRESPONDING TO THIS MINIMIZATION PROBLEM  USE THE  NOTATION RJL  EXTL XBARTJ TO OBTAIN THE WIENERHOPF  EQUATION R WBFF  RBFWHERE R  EXBFT1XBFHT1 RBF  EXBFT1XT ANDXBFT1  XBART1XBART2LDOTSXBARTMTITEM DETERMINE AN EXPRESSION FOR THE MINIMUM MEANSQUARED ERROR PM   MIN EFMT2ITEM SHOW THAT THE EQUATIONS FOR THE OPTIMAL WEIGHTS AND THE MINIMUM  MEANSQUARED ERROR CAN BE COMBINED INTO EM AUGMENTED WIENERHOPF  INDEXWIENERHOPF   EQUATIONS AS BEGINBMATRIX R0  RBFH  RBF  R ENDBMATRIXBEGINBMATRIX 1  WBFF ENDBMATRIX  BEGINBMATRIXPM   ZEROBF ENDBMATRIXITEM IF XT HAPPENS TO BE GENERATED BY AN ARM PROCESS DRIVEN  BY WHITE NOISE SUCH THAT IT IS THE OUTPUT OF A SYSTEM WITH TRANSFER  FUNCTION HZ  FRAC11SUMK1M AK ZKSHOW THAT THE PREDICTION COEFFICIENTS ARE WFK  AK AND HENCETHE COEFFICIENTS OF THE PREDICTION ERROR FILTER FMT ARE AFI  AIHENCE CONCLUDE THAT IN THIS CASE THE FORWARD PREDICTION ERRORFMT IS A EM WHITENOISE SEQUENCE  THE PREDICTIONERRORFILTER CAN THUS BE VIEWED AS A EM WHITENING FILTER FOR THE SIGNALXT ITEM NOW LET XHATTM  SUMI1M WBI XTI1BE THE EM BACKWARD PREDICTOR OF XTM USING THE DATA XTM1XTM2 LDOTS XT AND LET BMT  XTM  XHATTMBE THE BACKWARD PREDICTION ERROR  INDEXBACKWARD  PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORBACKWARDSHOW THAT THE WIENERHOPF EQUATIONS FOR THE OPTIMAL BACKWARD PREDICTOR CAN BE WRITTEN ASBEGINEQUATION R WBF  OVERLINERBFBLABELEQBACKPRED1ENDEQUATIONWHERE RBFB IS THE BACKWARD ORDERING OF RBF  DEFINED ABOVEITEM FROM REFEQBACKPRED1 SHOW THAT RH OVERLINEWBFBB  RBFWHERE WBFBB IS THE BACKWARD ORDERING OF WBFB  HENCE CONCLUDETHAT OVERLINEWBFBB  WBFFTHAT IS THE OPTIMAL BACKWARD PREDICTION COEFFICIENTS ARE THE REVERSEDCONJUGATED OPTIMAL FORWARD PREDICTION COEFFICIENTSENDENUMERATEITEM LET XT  08 XT1   NUTWHERE NUT IS A WHITENOISE ZEROMEAN UNIT VARIANCE NOISEPROCESS  WE WANT TO DETERMINE AN OPTIMAL PREDICTORBEGINENUMERATEITEM IF THE ORDER OF THE PREDICTOR IS 2 DETERMINE THE OPTIMAL  PREDICTOR XHATTITEM IF THE ORDER OF THE PREDICTOR IS 1 DETERMINE THE OPTIMAL  PREDICTOR XHATTENDENUMERATEITEM BF RANDOM VECTORS THE MEANSQUARED METHODS TO THIS POINT HAVE  BEEN FOR RANDOM SCALARS  SUPPOSE WE HAVE THE RANDOM VECTOR  APPROXIMATION PROBLEM YBF  C1 PBF1  C2 PBF2  CDOTS  CM XBFM  EBFIN WHICH WE DESIRE TO FIND AN APPROXIMATION YBF IN SUCH A WAY THATTHE NORM OF EBF IS MINIMIZED  LET US DEFINE AN INNER PRODUCTBETWEEN RANDOM VECTORS AS LA XBFYBF RA  TRACE EXBF YBFHBEGINENUMERATEITEM BASED UPON THIS INNER PRODUCT AND ITS INDUCED NORM DETERMINE A  SET OF NORMAL EQUATIONS FOR FINDING C1C2LDOTS CMITEM AS AN EXERCISE IN COMPUTING GRADIENTS USE THE FORMULA FOR THE  GRADIENT OF THE TRACE SEE APPENDIX REFAPPDXGRADIENT TO ARRIVE AT  THE SAME SET OF NORMAL EQUATIONSENDENUMERATEITEM MULTIPLE GAINSCALED VECTOR QUANTIZATION  INDEXVECTOR QUANTIZATION LET X AND Y BE VECTOR SPACES OF THE SAME  DIMENSIONALITY  SUPPOSE THAT THERE ARE TWO SETS OF VECTORS XC1  XC2 SUBSET X  LET YC BE THE SET OF VECTORS EM POOLED FROM  XC1 AND XC2 BY THE INVERTIBLE MATRICES T1 AND T2  RESPECTIVELY  THAT IS IF XBF IN XCI THEN YBF  TI XBF  IS A VECTOR IN YC  INDICATE THAT A VECTOR YBF IN YC CAME  FROM A VECTOR IN XCI BY A SUPERSCRIPT I SO YBFI IN YC  MEANS THAT THERE IS A VECTOR XBF IN XC SUCH THAT YBFI  TI  XBF  DISTANCES RELATIVE TO A VECTOR YBFI IN YC ARE BASED  UPON THE L2 NORM OF THE VECTORS OBTAINED BY MAPPING BACK TO  XCI SO THAT DYBFIYBF  YBFI  YBF  YBFI  YBFI T1YBFI  YBF  YBFI  YBFT WIYBFI  YBFWHERE WI  TITTI1  THIS IS A WEIGHTED NORM WITH THE WEIGHTINGDEPENDENT UPON THE VECTOR IN YC  WE DESIRE TO FIND A EM SINGLE VECTOR YBF0 IN Y THAT IS THE  BEST REPRESENTATION OF THE DATA POOLED FROM ALL THREE DATA SETS IN  THE SENSE THAT SUMYBF IN YC YBF  YBF0  SUMYBF1 IN YC  YBF1  YBF01  SUMYBF2 IN YC YBF2  YBF02IS MINIMIZED  SHOW THAT YBF0  Z1 RBFWHERE Z  SUMYBF1 IN YC W1  SUMYBF2 IN YC W2 QQUADTEXTANDQQUADRBF  SUMYBF1 IN YC W1  YBF1  SUMYBF2 IN YC W2YBF2HINT THIS IS PROBABLY EASIER USING GRADIENTS THAN TRYING TO IDENTIFYTHE APPROPRIATE INNER PRODUCTENDEXERCISESSECTIONCOMPARISON OF LEASTSQUARES AND MINIMUM MEANSQUARESLABELSECCMPIT IS INTERESTING TO CONTRAST THE METHOD OF LEASTSQUARES AND THEMETHOD OF MINIMUM MEANSQUARES BOTH OF WHICH ARE WIDELY USED INSIGNAL PROCESSING  FOR THE METHOD OF LEASTSQUARES WE MAKE THEFOLLOWING OBSERVATIONSBEGINENUMERATEITEM ONLY THE SEQUENCE OF DATA OBSERVED AT THE TIME OF THE ESTIMATE  IS USED IN FORMING THE ESTIMATEITEM DEPENDING UPON ASSUMPTIONS MADE ABOUT THE DATA BEFORE AND AFTER  THE OBSERVATION INTERVAL THE GRAMMIAN MATRIX MAY NOT BE TOEPLITZITEM NO STATISTICAL MODEL IS NECESSARILY ASSUMEDENDENUMERATEFOR THE METHOD OF MINIMUM MEANSQUARES WE MAKE THE FOLLOWINGOBSERVATIONSBEGINENUMERATEITEM A STATISTICAL MODEL FOR THE CORRELATIONS AND CROSSCORRELATIONS  IS NECESSARY  THIS MUST BE OBTAINED EITHER FROM EXPLICIT KNOWLEDGE  OF THE CHANNEL AND SIGNAL AS WAS SEEN IN EXAMPLE REFEXMEQ1 OR  ON THE BASIS OF THE MULTIVARIABLE DISTRIBUTION OF THE DATA AS WAS  SEEN IN EXAMPLE REFEXMMMSEPRED  IN THE ABSENCE OF SUCH  KNOWLEDGE IT IS COMMON TO ESTIMATE THE NECESSARY AUTOCORRELATION  AND CROSS CORRELATION VALUES  AN EXAMPLE OF AN ESTIMATE OF THE AUTOCORRELATION RN   EXKXBARKN USING THE DATA X1X2LDOTSXN ISBEGINEQUATION RHATN  FRAC1N SUMK1NN XK XBARKNLABELEQRHATNENDEQUATIONTHIS IS ACTUALLY A BIASED ESTIMATE OF RN SEE EXERCISEREFEXBIASCORR BUT IT HAS BEEN FOUND SEE EG CITEBOXJENKINS TOPRODUCE A LOWER VARIANCE WHEN THE LAG N IS CLOSE TO NIN ORDER FOR REFEQRHATN TO BE A REASONABLE ESTIMATE OF RNTHE RANDOM PROCESS XK MUST BE EM ERGODIC INDEXERGODIC SOTHAT THE TIME AVERAGE ASYMPTOTICALLY APPROACHES THE ENSEMBLE AVERAGETHIS ASSUMPTION OF ERGODICITY IS USUALLY MADE TACITLY BUT IT IS VITALWHEN THE DATA SEQUENCE USED TO COMPUTE THE ESTIMATE OF THE CORRELATIONPARAMETERS IS THE SAME AS THE DATA SEQUENCE FOR WHICH THE FILTERCOEFFICIENTS ARE COMPUTED THE MINIMUM MEANSQUARED ERROR TECHNIQUE ISESSENTIALLY THE SAME AS THE LEASTSQUARES TECHNIQUEITEM COMMONLY THE COEFFICIENTS OF THE MMS TECHNIQUE ARE COMPUTED  USING A SEPARATE SET OF DATA WHOSE STATISTICS ARE EM ASSUMED TO  BE THE SAME AS THOSE OF THE REAL DATA SET OF INTEREST  THIS SET OF  DATA IS USED AS A EM TRAINING SET INDEXTRAINING SET TO FIND  THE AUTOCORRELATION FUNCTIONS AND THE FILTER COEFFICIENTS  PROVIDED  THAT THE TRAINING DATA DOES HAVE THE SAME OR VERY SIMILAR  STATISTICS AS THE DATA SET OF INTEREST THIS WORKS WELL  HOWEVER  IF THE TRAINING DATA IS SIGNIFICANTLY DIFFERENT FROM THE DATA SET OF  INTEREST FINDING THE OPTIMUM FILTER COEFFICIENTS CAN ACTUALLY LEAD  TO POOR PERFORMANCE BECAUSE THE BEST SOLUTION TO THE  WRONG PROBLEM IS USEDITEM WE ALSO NOTE THAT THE TRUE GRAMMIAN MATRIX R USED IN  PREDICTION AND OPTIMAL FIR FILTERING PROBLEMS IS ALWAYS A TOEPLITZ  MATRIX AND HENCE FAST ALGORITHMS APPLY TO FINDING THE  COEFFICIENTSENDENUMERATEIN SECTION REFSECRLS WE EXAMINE HOW THE COEFFICIENTS OF THE LSFILTER CAN BE UPDATED ADAPTIVELY SO THAT THE COEFFICIENTS AREMODIFIED AS NEW DATA ARRIVES  IN SECTION REFSECLMS WE DEVELOP ANALGORITHM SO THAT THE COEFFICIENTS OF THE MMS FILTER CAN BE UPDATEDADAPTIVELY BY APPROXIMATING THE EXPECTATION  THESE TWO CONCEPTS FORMTHE HEART OF ADAPTIVE FILTERING THEORYBEGINEXERCISES  ITEM LABELEXBIASCORR FOR THE ESTIMATED AUTOCORRELATION RHATN  FRAC1N SUMK1NN XK XBARKNBEGINENUMERATEITEM TAKE THE EXPECTATION ERHATN AND SHOW THAT IT IS NOT  EQUAL TO RN THE TRUE VALUE OF THE AUTOCORRELATIONITEM DETERMINE A SCALING FACTOR TO MAKE THE RHATN AN UNBIASED  ESTIMATEITEM WRITE A SC MATLAB FUNCTION THAT COMPUTES RHATN FROM  REFEQRHATNENDENUMERATEENDEXERCISESINPUTDETESTDIRFREQFILTSECTIONA DUAL APPROXIMATION PROBLEMLABELSECDUALAPPROXINDEXDUAL APPROXIMATION THE APPROXIMATION PROBLEMS WE HAVE SEEN UPTILL NOW HAVE SELECTED A POINT FROM A FINITEDIMENSIONAL SUBSPACE OFTHE HILBERT SPACE OF THE PROBLEM  IN EACH CASE BECAUSE THE SOLUTIONWAS IN A FINITEDIMENSIONAL SUBSPACE SOLVING AN MATSIZEMMSYSTEM OF EQUATIONS WAS SUFFICIENT  IN SOME APPROXIMATION PROBLEMSTHE SUBSPACE IN WHICH THE SOLUTION LIES IS NOT FINITE DIMENSIONAL SOA SIMPLE FINITE SET OF EQUATIONS CANNOT BE SOLVED TO OBTAIN THESOLUTION  THERE ARE SOME PROBLEMS HOWEVER IN WHICH A FINITE SET OFCONSTRAINTS PROVIDES US WITH SUFFICIENT INFORMATION TO SOLVE THEPROBLEM FROM A FINITE SET OF EQUATIONSWE BEGIN WITH A DEFINITIONBEGINDEFINITION  LET M BE A SUBSPACE OF A LINEAR SPACE S AND LET X0 IN S  THE SET V  X0  M IS SAID TO BE A EM TRANSLATION OF M BY  X0  THIS TRANSLATION IS CALLED A BF LINEAR  VARIETY INDEXLINEAR VARIETYENDDEFINITIONA LINEAR VARIETY IS NOT IN GENERAL A SUBSPACEBEGINEXAMPLE  LET M  000010 IN THE VECTOR SPACE GF23  INTRODUCED IN EXAMPLE REFEXMVS1 AND LET XBF0  111 IN  S  THEN XBFM  111101IS A LINEAR VARIETYENDEXAMPLEA VERSION OF THE ORTHOGONALITY THEOREM APPROPRIATE FOR LINEARVARIETIES IS ILLUSTRATED IN FIGURE REFFIGLINVAR  LET V  XBF0 M BE A CLOSED LINEAR VARIETY IN A HILBERT SPACE H  THEN THERE ISA EM UNIQUE VECTOR VBF0 IN V OF MINIMUM NORM  THE MINIMIZINGVECTOR VBF0 IS ORTHOGONAL TO M  THIS RESULT IS AN IMMEDIATECONSEQUENCE OF THE PROJECTION THEOREM FOR HILBERT SPACES SIMPLYTRANSLATE THE VARIETY AND THE ORIGIN BY XBF0BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLINVAR1    CAPTIONMINIMUM NORM TO A LINEAR VARIETY    LABELFIGLINVAR  ENDCENTERENDFIGURELET S BE A HILBERT SPACE  GIVEN A SET OF LINEARLY INDEPENDENTVECTORS YBF1YBF2LDOTSYBFM IN S LET M LSPANYBF1YBF2LDOTSYBFM  THE SET OF XBF IN S SUCH THAT BEGINALIGNEDLA XBFYBF1RA  0 LA XBFYBF2RA  0 VDOTS LA XBFYBFMRA  0 ENDALIGNEDIS A SUBSPACE WHICH BECAUSE OF THESE INNERPRODUCT CONSTRAINTS MUSTBE MPERP  SUPPOSE NOW WE HAVE PROBLEM IN WHICHTHERE ARE INNERPRODUCT CONSTRAINTS OF THE FORMBEGINEQUATIONLABELEQDUAL1BEGINSPLITLA XBFYBF1 RA  A1 LA XBFYBF2 RA  A2   VDOTS LA XBFYBFM RA  AMENDSPLITENDEQUATIONIF WE CAN FIND ANY POINT XBFXBF0 THAT SATISFIES THE CONSTRAINTSIN REFEQDUAL1 THEN FOR ANY VBF IN MPERP XBF0 VBFALSO SATISFIES THE CONSTRAINTS  HENCE THE SPACE OF SOLUTIONS OFREFEQDUAL1 IS THE LINEAR VARIETY V  XBF0  MPERP  ALINEAR VARIETY V SATISFYING THE M CONSTRAINTS IN REFEQDUAL1IS SAID TO HAVE EM CODIMENSION M INDEXCODIMENSION SINCE THEORTHOGONAL COMPLEMENT OF THE SUBSPACE MPERP PRODUCING IT HASDIMENSION MBEGINEXAMPLE  IN RBB3 LET YBF1  100 AND YBF2  010  AND LET  M  LSPANYBF1YBF2  THE SET OF POINTS SUCH THAT LA XBFYBF1RA  0 QQUADQQUADLA XBFYBF2 RA  0IS LSPAN001  MPERPNOW FOR THE CONSTRAINTSLA XBFYBF1RA  3 QQUADQQUADLA XBFYBF2 RA  4OBSERVE THAT IF XBF  34S FOR ANY S IN RBB THEN THECONSTRAINTS ARE SATISFIED  THE SET V  340  MPERP IS ALINEAR VARIETY OF CODIMENSION 2ENDEXAMPLEWE ARE NOW IN A POSITION TO STATE THE MINIMIZATION PROBLEM  BEGINTHEOREM LABELTHMDUALAPPROX DUAL APPROXIMATION LET  YBF1YBF2LDOTSYBFM BE   LINEARLY INDEPENDENT IN A HILBERT SPACE S AND LET M   LSPANYBF1LDOTSYBFM  THE ELEMENT XBFIN S SATISFYINGBEGINEQUATIONBEGINSPLITLA XBFYBF1 RA  A1 LA XBFYBF2 RA  A2  VDOTS  LA XBFYBFM RA  AMENDSPLITLABELEQDUAL2ENDEQUATIONWITH MINIMUM NORM LIES IN M SPECIFICALLY XBF  SUMI1M CI YBFIWHERE THE COEFFICIENTS IN THIS LINEAR COMBINATION SATISFYBEGINEQUATIONBEGINBMATRIX LA YBF1YBF1RA  LA YBF2YBF1RA  CDOTS  LA YBFMYBF1RA LA YBF1YBF2RA  LA YBF2YBF2RA  CDOTS  LA YBFMYBF2RA VDOTS LA YBF1YBFMRA  LA YBF2YBFMRA  CDOTS  LA YBFMYBFMRA ENDBMATRIXBEGINBMATRIX C1  C2  VDOTS  CM ENDBMATRIX BEGINBMATRIX A1 A2  VDOTS  AM ENDBMATRIXLABELEQDUAL3ENDEQUATIONENDTHEOREMBEGINPROOF  BY THE DISCUSSION ABOVE THE SOLUTION LIES IN THE LINEAR VARIETY V   XBF0  MPERP FOR SOME XBF0  FURTHERMORE THE OPTIMAL SOLUTION  XBF0 IS ORTHOGONAL TO MPERP SO THAT XBF0 IN MPERPPERP   M  THUS XBF0 IS OF THE FORM XBF0  SUMI1M CI YBFITAKING INNER PRODUCTS OF THIS EQUATION WITH YBF1YBF2LDOTSYBFM AND RECOGNIZING THAT FOR THE SOLUTION LA XBF0 YBFIRA AI WE OBTAIN THE SET OF EQUATIONS IN REFEQDUAL3ENDPROOFBEGINEXAMPLE  FOR THE LINEAR VARIETY OF THE PREVIOUS PROBLEM LET US FIND THE  SOLUTION OF MINIMUM NORM  USING REFEQDUAL3 WE FINDX  340TO BE THE MINIMUM NORM SOLUTION SATISFYING THE CONSTRAINTSENDEXAMPLEBEGINEXAMPLE  WE EXAMINE HERE A PROBLEM IN WHICH THE SOLUTION SPACE IS INFINITE  DIMENSIONAL  SUPPOSE WE HAVE AN LTI SYSTEM WITH CAUSAL IMPULSE  RESPONSE HT  E2T  3E4T IN WHICH THE INITIAL CONDITIONS  ARE Y0  0 AND YDOT0  0  WE DESIRE TO DETERMINE AN INPUT  SIGNAL XT SO THAT THE OUTPUT YT  XTHT SATISFIES THE  CONSTRAINTSY1 1 QQUADQQUADINT01 YT DT  0IN SUCH A WAY THAT THE INPUT ENERGY INT01 XT2 DTIS MINIMIZED  WRITING THE CONVOLUTION INTEGRAL FOR THE FIRST OUTPUTTHE FIRST CONSTRAINT CAN BE WRITTEN INT01 E21TAU  3E41TAUXTAU DTAU  1USING THE INNER PRODUCT LA FGRA  INT01 FTAUGTAU DTTHE FIRST CONSTRAINT CAN BE WRITTEN AS LA XY1RA  1WHERE Y1TAU  E21TAU  3E41TAU  H1TAUTHE SECOND CONSTRAINT CAN BE WRITTEN USING THE INTEGRAL OF THE IMPULSERESPONSE SEE EXERCISE REFEXINTSYSRESP KT  INT0T HTAUDT  FRAC54  FRAC34E4T FRAC12 E2TTHEN THE SECOND CONSTRAINT IS LA XY2RA  0WHERE  Y2TAU  FRAC54  FRAC34 E41TAU  FRAC12E21TAU  K1TAUTHE SOLUTION X0T MUST LIE IN THE SPACE SPANNED BY Y1 ANDY2 X0  C1 Y1T  C2 Y2TTHEN THE EQUATION REFEQDUAL3 BECOMES BEGINBMATRIXLA Y1Y1RA  LA Y1Y2RA  LA Y1Y2 RA  LA Y2Y2 RA ENDBMATRIXBEGINBMATRIXA1  A2 ENDBMATRIX BEGINBMATRIX 236756 0682808 0682808   0818254ENDBMATRIXBEGINBMATRIX C1  C2  ENDBMATRIX  BEGINBMATRIX 0  1ENDBMATRIXWHICH HAS SOLUTION BEGINBMATRIXA1A2 ENDBMATRIXT  BEGINBMATRIX0464166  160945 ENDBMATRIX DUALMINMMAENDEXAMPLESECTIONMINIMUMNORM SOLUTION OF UNDERDETERMINED EQUATIONSLABELSECLS2THE SOLUTION TO THE DUAL APPROXIMATION PROBLEM PROVIDES A METHOD OFFINDING A LEASTSQUARES SOLUTION TO AN UNDERDETERMINED SET OFEQUATIONSBEGINEXAMPLE  SUPPOSE THAT WE ARE TO SOLVE THE SET OF EQUATIONSBEGINEQUATION BEGINBMATRIX12 3  541 ENDBMATRIX BEGINBMATRIXX1X2  X3ENDBMATRIX  BEGINBMATRIX 4  6 ENDBMATRIXLABELEQMINNORMSOL1ENDEQUATIONONE SOLUTION IS XBF  BEGINBMATRIX1  2  3 ENDBMATRIXHOWEVER OBSERVE THAT THE VECTOR VBF  111T IS IN THENULLSPACE OF A SO THAT A VBF  0 ANY VECTOR OF THE FORM BEGINBMATRIX1  2  3 ENDBMATRIX  T BEGINBMATRIX 1  1 1 ENDBMATRIXFOR T IN RBB IS ALSO A SOLUTION TO REFEQMINNORMSOL1ENDEXAMPLEWHEN SOLVING M EQUATIONS WITH N UNKNOWNS WITH M  N UNLESS THEEQUATIONS ARE INCONSISTENT AS IN THE EXAMPLE BEGINBMATRIX 123  2  4 6ENDBMATRIXBEGINBMATRIXX1X2  X3 ENDBMATRIX BEGINBMATRIX4  7 ENDBMATRIXTHERE WILL BE AN INFINITE NUMBER OF SOLUTIONSLET XBF BE A SOLUTION OF AXBF  BBF WHERE A IS ANMATSIZEMN MATRIX WITH MN AND LET N  NULLSPACEATHEN IF XBF0 IS A SOLUTION TO AXBF  BBF SO IS ANY VECTOR OFTHE FORM XBF0  NBF WHERE NBF IN N  IF THE NULLSPACE IS NOTTRIVIAL A VARIETY OF SOLUTIONS ARE POSSIBLE  IN ORDER TO HAVE AWELLDETERMINED ALGORITHM FOR UNIQUELY SOLVING THE PROBLEM SOMECRITERION MUST BE ESTABLISHED REGARDING WHICH SOLUTION IS DESIRED  AREASONABLE CRITERION IS TO FIND THE SOLUTION XBF OF SMALLEST NORMTHAT IS WE WANT TO BEGINALIGNEDTEXTMINIMIZE  XBF TEXTSUBJECT TO   A XBF  BBFENDALIGNEDTHE MINIMUM NORM SOLUTION IS APPEALING FROM A NUMERIC STANDPOINTBECAUSE REPRESENTATIONS OF SMALL NUMBERS ARE USUALLY EASIER THANREPRESENTATIONS OF LARGE NUMBERS  IT ALSO LEADS TO A UNIQUE SOLUTIONTHAT CAN BE COMPUTED USING THE FORMULATION OF THE DUAL PROBLEM OF THEPREVIOUS SECTIONLET US WRITE A IN TERMS OF ITS ROWS AS A  BEGINBMATRIX YBF1H  YBF2H   VDOTS   YBFMHENDBMATRIXTHEN WE OBSERVE THAT THE EQUATION AXBF  BBF IS EQUIVALENT TO BEGINALIGNEDYBFH1 XBF  B1 YBFH2 XBF  B2 VDOTS YBFHM XBF  BMENDALIGNEDOUR CONSTRAINT EQUATION THEREFORE CORRESPONDS TO M INNERPRODUCTCONSTRAINTS OF THE SORT SHOWN IN REFEQDUAL1  BY THEOREMREFTHMDUALAPPROX THE MINIMUMNORM SOLUTION MUST BE OF THE FORMBEGINEQUATION XBF  SUMI1M CI YBFILABELEQDUAL4ENDEQUATIONWHERE THE CI ARE THE SOLUTION TO REFEQDUAL3WE CAN WRITE REFEQDUAL4 ASBEGINEQUATION XBF  AH CBFLABELEQDUAL5ENDEQUATIONWHERE AH  BEGINBMATRIXYBF1  YBF2  CDOTS  YBFMENDBMATRIXFURTHERMORE IN MATRIX NOTATION WE CAN WRITE REFEQDUAL3 IN THEFORM AAHCBF  BBFPROVIDED THAT THE ROWS ARE LINEARLY INDEPENDENT THE MATRIX AAH ISINVERTIBLE AND WE CAN SOLVE FOR CBF AS CBF  AAH1BBFSUBSTITUTING THIS INTO REFEQDUAL5 WE OBTAIN THE MINIMUMNORMSOLUTION BEGINEQUATIONXBF  AHAAH1 BBFLABELEQPSEUDOINV2ENDEQUATIONBEGINEXAMPLE  THE MINIMUM NORM SOLUTION TO REFEQMINNORMSOL1 FOUND USING  REFEQPSEUDOINV2 IS XBF  BEGINBMATRIX1  0  1 ENDBMATRIXENDEXAMPLETHE MATRIX AHAAH1 IS A PSEUDOINVERSE INDEXPSEUDOINVERSEOF THE MATRIX ABEGINEXERCISESITEM LABELEXINTSYSRESP LET HT BE THE IMPULSE RESPONSE OF A  SYSTEM AND LET YT  XTHT  SHOW THAT INT0T YTDT  YTKTWHERE KT IS THE INTEGRAL OF THE IMPULSE RESPONSE KT  INT0T HTDTITEM A SYSTEM IS KNOWN TO HAVE IMPULSE RESPONSE HT  3E2T   4E5T AND IS INITIALLY RELAXED INITIAL CONDITIONS ARE ZERO  DETERMINE AN INPUT XT SO THAT THE OUTPUT SATISFIES THE  CONDITIONS Y2  2 QQUAD TEXTANDQQUAD INT02 YTDT  3IN SUCH A WAY THAT THE INPUT ENERGY XT2 IS MINIMIZED  ITEM LET HT  02T  304T FOR K GEQ 0 BE THE IMPULSE    RESPONSE OF A DISCRETETIME SYSTEM WITH ZERO INITIAL CONDITIONS    IT IS DESIRED TO DETERMINE A CAUSAL INPUT SEQUENCE XT SUCH    THAT THE OUTPUT YT  HTFT SATISFIES THE CONSTRAINTS BEGINALIGNEDY10  5SUMJ010 YJ  2ENDALIGNEDSUCH THAT THE INPUT ENERGY SUMK010 XT2 IS MINIMIZEDFORMULATE THIS AS A DUAL APPROXIMATION PROBLEM AND FIND THE MINIMIZINGSEQUENCE XTITEM CITELUENBERGER1969  USING THE PROJECTION THEOREM SOLVE THE  FINITE DIMENSIONAL PROBLEM BEGINALIGNEDTEXTMINIMIZE   XBFT Q XBF TEXTSUBJECT TO   A XBF  BBFENDALIGNEDWHERE XBF IN RBBN Q IS A POSITIVEDEFINITE SYMMETRIC MATRIXAND A IS A MATSIZEMN MATRIX WITH M  NITEM CITELUENBERGER1969  LET XBF BE A VECTOR IN A HILBERT SPACE  S AND LET  XBF1 XBF2LDOTSXBFN AND   YBF1YBF2LDOTS YBFM BE SETS OF LINEARLY INDEPENDENT  VECTORS IN S  WE DESIRE TO MINIMIZE  XBF  XBFHAT WHILE  SATISFYING XBF IN M  LSPAN XBF1XBF2LDOTS XBFMAND LA XBFHAT YBFIRA  CI I12LDOTS MBEGINENUMERATEITEM FIND EQUATIONS FOR THE SOLUTION WHICH ARE SIMILAR TO THE NORMAL  EQUATIONS ITEM GIVE A GEOMETRIC INTERPRETATION OF THE EQUATIONSENDENUMERATEITEM CITELUENBERGER1969 CONSIDER THE PROBLEM OF FINDING THE VECTOR  XBF OF MINIMUM NORM SATISFYING LA XBFYBFIRA GEQ CI FOR  I12LDOTS N WHERE THE YBFIS ARE LINEARLY INDEPENDENT  BEGINENUMERATE  ITEM SHOW THAT THIS PROBLEM HAS A UNIQUE SOLUTION  ITEM SHOW THAT A NECESSARY AND SUFFICIENT CONDITION THAT XBF  SUMI1N AI YBFIIS A SOLUTION IS THAT THE VECTOR ABF WITH COMPONENTS AI SATISFY RT ABF GEQ CBF QQUAD TEXTANDQQUAD ABF GEQ ZEROBFAND THAT AI  0 IF LA XBFYBFIRA  CI  R IS THE GRAMMIANMATRIX OF YBF1YBF2LDOTS YBFN  ENDENUMERATEENDEXERCISESINPUTLINALGDIRWLS   IRLS SECTIONSECTIONSIGNAL TRANSFORMATION AND GENERALIZED FOURIER SERIESLABELSECGFSMUCH OF THE TRANSFORM THEORY EMPLOYED IN SIGNAL PROCESSING ISENCOMPASSED BY REPRESENTATIONS IN AN APPROPRIATE LINEAR VECTOR SPACETHE SET OF BASIS FUNCTIONS FOR THE TRANSFORMATION IS CHOSEN SO THATTHE COEFFICIENTS CONVEY DESIRED INFORMATION ABOUT THE SIGNAL  BYDETERMINING THE BASIS FUNCTIONS APPROPRIATELY DIFFERENT INFORMATIONCAN BE EXTRACTED FROM A SIGNAL BY FINDING A REPRESENTATION OF THESIGNAL IN THE BASISIN THIS SECTION WE ARE LARGELY BUT NOT ENTIRELY INTERESTED INAPPROXIMATING CONTINUOUSTIME FUNCTIONS  THE METRIC SPACE IS L2AND WE DEAL WITH AN INFINITE NUMBER OF BASIS FUNCTIONS SO SOMEWHATMORE CARE IS NEEDED THAN IN THE PREVIOUS SECTIONS OF THIS CHAPTERFINDING THE BEST REPRESENTATION IN AN L2 NORM SENSE OF A FUNCTIONXT AS XT APPROX SUMI0M CI PITWHERE PIT IS A SET OF BASIS FUNCTIONS IS THE APPROXIMATIONPROBLEM WE HAVE SEEN ALREADY MANY TIMES  IF THE BASIS FUNCTIONS AREORTHONORMAL THE COEFFICIENTS WHICH MINIMIZE X  SUMI0M CIPI2 CAN BE FOUND AS CI  LA XPIRA THE SET OF COEFFICIENTSCII12LDOTSM PROVIDES THE BEST REPRESENTATION IN THELEASTSQUARES SENSE OF X  THE MINIMUM SQUARED ERROR OF THE SERIESREPRESENTATION IS  X  SUMI1M CI PI2  X2  SUMI1M LA XPIRA2SINCE THE ERROR IS NEVER NEGATIVE IT FOLLOWS THATBEGINEQUATION SUMI1M CI2  SUMI1M LA XPIRA2 LEQ X2LABELEQBESSELINEQENDEQUATIONTHIS INEQUALITY IS KNOWN AS EM BESSELS INEQUALITYTHE FUNCTION SUMI1M CI PI OBTAINED AS A BEST L2APPROXIMATION OF XT IS SAID TO BE THE EM PROJECTION OF XTONTO THE SPACE SPANNED BY P1P2LDOTSPM  THIS MAY BEWRITTEN AS  XPROJP1P2LDOTSPMTASSUME THAT X AND  PI ARE IN SOME HILBERT SPACE H  IF THESET OF BASIS FUNCTIONS  PI IS INFINITE WE CAN TAKE THE LIMITIN REFEQBESSELINEQ AS M RIGHTARROW INFTY  THEREPRESENTATION OF THIS LIMIT IS THE INFINITE SERIES YT  SUMI1INFTY CI PITSINCE YMT  SUMI1M CI PITIS A CAUCHY SEQUENCE AND THE HILBERT SPACE IS COMPLETE WE CONCLUDETHAT YT IS IN THE HILBERT SPACE  FOR ANY ORTHONORMAL SET PI THE BEST APPROXIMATION OF X IN THE L2 SENSE IS THEFUNCTION Y  WE NOW WANT TO ADDRESS THE QUESTION OF WHEN XY FORAN ARBITRARY X IN H  WE MUST FIRST POINT OUT THAT BY THEEQUALITY XY WHAT WE MEAN IS THAT  XY  0WHERE THE NORM IS THE L2 NORM SINCE WE ARE DEALING WITH A HILBERTSPACE  FUNCTIONS THAT DIFFER ON A SET OF MEASURE ZERO ARE EQUALIN THE SENSE OF THE L2 NORM  THUS EQUAL DOES NOT NECESSARILYMEAN POINTFORPOINT EQUAL AS DISCUSSED IN SECTION REFSECMETTECHWE NOW DEFINE A CONDITION UNDER WHICHIT IS POSSIBLE TO REPRESENT EVERY X USING THE BASIS SET PIBEGINDEFINITION  AN ORTHONORMAL SET PII12LDOTSINFTY IN A HILBERT SPACE  S IS BF COMPLETEFOOTNOTETHIS REFERS TO COMPLETENESS OF THE    SET OF FUNCTIONS WHICH CONCERNS THE REPRESENTATIONAL ABILITY    OF THE FUNCTIONS NOT THE COMPLETENESS OF THE SPACE WHICH IS USED    TO DESCRIBE THE FACT THAT ALL CAUCHY SEQUENCES CONVERGE  SOME    AUTHORS USE TOTAL IN PLACE OF COMPLETE HERE INDEXTOTAL      SETSEECOMPLETE SET INDEXCOMPLETE SET IF X  SUMI1INFTY LA XPIRA PIFOR EVERY X IN SENDDEFINITIONBEGINEXAMPLE  BY MEANS OF A SIMPLE COUNTEREXAMPLE IT IS STRAIGHTFORWARD TO SHOW  THAT SIMPLY HAVING AN INFINITE SET OF ORTHONORMAL FUNCTIONS IS NOT  SUFFICIENT TO ESTABLISH COMPLETENESS  IN L202PI CONSIDER  THE FUNCTION XT  COS T  AN INFINITE SET OF ORTHOGONAL  FUNCTIONS IS T   PNT  SINNT N12LDOTS  IN THE  GENERALIZED FOURIER SERIES REPRESENTATION XHATT  SUMI1INFTY CI PITWE FIND THAT THE COEFFICIENTS ARE PROPORTIONAL TO LA COS T SIN NTRA  INT02PI COST SINNTDT  0HENCE XHATT0 WHICH IS NOT A GOOD REPRESENTATION  WE CONCLUDETHAT THE SET IS NOT COMPLETEENDEXAMPLESOME RESULTS REGARDING COMPLETENESS ARE EXPRESSED IN THE FOLLOWINGTHEOREM WHICH WE STATE WITHOUT PROOFBEGINTHEOREM CITEKEENER  A SET OF ORTHONORMAL FUNCTIONS PI I12LDOTS IS COMPLETE  IN AN INNER PRODUCT SPACE S WITH INDUCED NORM IF ANY OF THE FOLLOWING EQUIVALENT STATEMENTS HOLDS  BEGINENUMERATE  ITEM FOR ANY X IN S X  SUMI1INFTY LA XPI RA PIITEM FOR ANY EPSILON  0 THERE IS AN N  INFTY SUCH THAT FOR  ALL N GEQ N  X  SUMI1N LA XPIRA PI EPSILONIN OTHER WORDS WE CAN APPROXIMATE ARBITRARILY CLOSELY ITEM PARSEVALS EQUALITY HOLDS X2  SUMI1INFTY LA  XPIRA2 FOR ALL X IN SITEM IF LA XPIRA  0 FOR ALL I THEN X0  THIS WAS SHOWN  TO FAIL IN THE LAST EXAMPLEITEM THERE IS NO NONZERO FUNCTION F IN S FOR WHICH THE SET  PII12LDOTS CUP F FORMS AN ORTHOGONAL SET  ENDENUMERATEENDTHEOREMFOR A FINITEDIMENSIONAL SPACE S OF DIMENSION M TO HAVE MLINEARLY INDEPENDENT FUNCTIONS PK K12LDOTSM IS SUFFICIENTFOR COMPLETENESSWHEN PI IS A COMPLETE BASIS SET THEN THE SEQUENCE C1C2LDOTS COMPLETELY DESCRIBES X THERE IS A ONETOONERELATIONSHIP BETWEEN X AND C1C2LDOTS  EXCEPT THAT XIS ONLY UNIQUE UP TO A SET OF MEASURE ZERO  WE SOMETIMES SAYTHAT THE SEQUENCE C1C2LDOTS IS THE BF TRANSFORM OR THEBF GENERALIZED FOURIER SERIES INDEXFOURIER SERIESGENERALIZED OFX  INDEXLEFTRIGHTARROWWRITING CBF  C1C2LDOTSWE CAN REPRESENT THE TRANSFORM RELATIONSHIP AS X LEFTRIGHTARROW CBFWE CAN DEFINE EM DIFFERENT TRANSFORMATIONS DEPENDING UPON THE SETOF ORTHONORMAL BASIS FUNCTIONS WE CHOOSE  SINCE EACH COEFFICIENT INTHE TRANSFORM IS A PROJECTION OF X ONTO THE BASIS FUNCTION THETRANSFORM COEFFICIENT PI DETERMINES HOW MUCH OF PI IS IN XIF WE WANT TO LOOK FOR PARTICULAR FEATURES OF A SIGNAL ONE WAY IS TODESIGN A SET OF ORTHOGONAL BASIS FUNCTIONS THAT HAVE THOSE FEATURESAND COMPUTE A TRANSFORM USING THOSE SIGNALSIF PII12LDOTS IS A COMPLETE SET THERE IS NO ERROR IN THEREPRESENTATION SO BESSELS INEQUALITY REFEQBESSELINEQINDEXBESSELS INEQUALITY INDEXINEQUALITIESBESSELSBECOMES AN EQUALITYBEGINEQUATIONX2  SUMI1INFTY CI2LABELEQPARSEVALENDEQUATIONTHIS RELATIONSHIP IS KNOWN AS EM PARSEVALS EQUALITYINDEXPARSEVALS EQUALITY IT SHOULD BEFAMILIAR IN VARIOUS SPECIAL CASES TO SIGNAL PROCESSORS  WE CAN WRITETHIS AS X  CBFWHERE THE NORM ON THE LEFT IS THE L2 NORM IF X IS A FUNCTIONAND THE NORM ON THE RIGHT IS THE L2 NORMFOR TRANSFORMATIONS USING ORTHONORMAL BASIS SETS THE ANGLES ARE ALSOPRESERVEDBEGINLEMMA  IF X AND Y HAVE A GENERALIZED FOURIER SERIES REPRESENTATION  USING SOME ORTHONORMAL BASIS SET PII12LDOTS IN A HILBERT  SPACE S WITH X LEFTRIGHTARROW CBF QQUADTEXTANDQQUADY LEFTRIGHTARROW BBFTHENBEGINEQUATION LA XY RA  LA CBF BBF RALABELEQPRESERANGLEIPENDEQUATIONENDLEMMABEGINPROOF  WE CAN WRITE X  SUMI1INFTY CI PI QQUAD TEXTANDQQUAD   Y  SUMI1INFTY BI PITHENBEGINALIGNLA XY RA  LA SUMI1INFTY CI PI SUMJ1INFTY BJ PJRA  LABELEQANGLEPROOF1  SUMI1INFTY CI BI  LA CBFBBFRA NONUMBERENDALIGNWHERE THE CROSS PRODUCTS IN THE INNER PRODUCT INREFEQANGLEPROOF1 ARE ZERO BECAUSE OF ORTHOGONALITYENDPROOFBEGINEXAMPLE LABELEXMFS BF FOURIER SERIES  INDEXFOURIER    SERIES THE SET OF  FUNCTIONS WHICH ARE PERIODIC ON 02PI CAN BE REPRESENTED USING  THE SERIES FT  SUMNINFTYINFTY CN FRAC1SQRT2PI EJN TTHE BASIS FUNCTIONS PNT  EJ N TSQRT2PI ARE ORTHONORMALSINCEINT02PI EJNT EJMT  DT  BEGINCASES0  N NEQ M 2PI  N  MENDCASESTHEN FROM REFEQPROJ4CN  FRAC1SQRT2PI INT02PI FT EJNTDTBY PARSEVALS RELATIONSHIP WE HAVE INT02PIFT2 DT   SUMN CN2MORE COMMONLY WE USE THE NONNORMALIZED BASIS FUNCTIONS YNT EJNT SO THE SERIES IS FT  SUMN BN EJNTABSORBING THE NORMALIZING CONSTANT INTO THE COEFFICIENT AS BN  FRAC12PI INT02PI FT EJNT DTIN THIS CASE PARSEVALS RELATIONSHIP MUST BE NORMALIZED AS INT02PI FT2  DT  FRAC12PI SUMI BI2MORE GENERALLY FOR A FUNCTION PERIODIC WITH PERIOD T0 WE HAVE THEFAMILIAR FORMULAS FT  SUMN BN EJNOMEGA0 TWHERE OMEGA0  2PIT0 AND BN  FRAC1T0INT0T0 FTEJNOMEGA0 T DTENDEXAMPLEBEGINEXAMPLE  DISCRETE FOURIER TRANSFORM DFT INDEXDISCRETE FOURIER TRANSFORM DFT A DISCRETETIME SEQUENCE XTT01LDOTSN1 IS TO BE  REPRESENTED AS A LINEAR COMBINATION OF THE FUNCTIONS PKT   1SQRTNEJ2PI TK N BY XT  FRAC1SQRTN SUMK0N1 CK EJ2PI TKNTHE INNER PRODUCT IN THIS CASE IS LA XTYT RA  SUMK0N1 XT YBARTIT CAN BE SHOWN SEE EXERCISE REFEXORTHOGDFT THAT THE SET OFBASIS FUNCTIONS PKT ARE ORTHOGONAL WITH LA PKTPLT RA BEGINCASES1  KBMOD L PMODN 0  TEXTOTHERWISEENDCASESTHE COEFFICIENTS ARE THEREFORE COMPUTED BY CK  FRAC1SQRTNSUMT0N1 XT EJ2PI TKNMORE COMMONLY WE USE THE EM NONNORMALIZED BASIS FUNCTIONS EJ2PI  TKN AND SHIFT ALL OF THE NORMALIZATION INTO THE RECONSTRUCTIONFORMULA  THEN WE HAVE XT  FRAC1N SUMK0N1 DK EJ2PI NKNAND DK  SUMT0N1 XT EJ2PI TKNWHICH IS THE USUAL FOURIER TRANSFORM PAIR PARSEVALSRELATIONSHIP UNDER THIS NORMALIZATION IS SUMT0N1 XT2  FRAC1N SUMK0N1 DK2ENDEXAMPLEBEGINEXERCISESITEM LABELEXORTHOGDFT SHOW THAT THE SET OF FUNCTIONS DEFINED BY PKT FRAC1SQRTN EJ2PI KTNARE ORTHONORMAL WITH RESPECT TO THE INNER PRODUCT LA XTYTRA  SUMK0N1 XTYBARTITEM LET  FT  ET2BE PERIODIC ON 0PIBEGINENUMERATEITEM FIND THE FOURIER SERIES COEFFICIENTS OF THIS FUNCTIONITEM FIND THE SUM OF THE SERIES SUMN LEFTFRAC2PI FRAC11 16N2RIGHT2HINT USE PARSEVALS THEOREMITEM IN THIS PROBLEM WE WILL SHOW THAT THE SET OF CONTINUOUS  FUNCTIONS IS COMPLETE WITH RESPECT TO THE UNIFORM SUP NORM  LET   FNT BE A CAUCHY SEQUENCE OF CONTINUOUS FUNCTIONS  LET  FT BE THE POINTWISE LIMIT OF FNT  FOR ANY EPSILON  0 LET N BE CHOSEN SO THAT MAX FNTFMT LEQ  EPSILON3  SINCE FKT IS CONTINUOUS THERE IS A DELTA SUCH  THAT FTDELTA  FT  EPSILON3  FROM THIS CONCLUDE THAT  FTDELTA  FT  EPSILONAND HENCE THAT FT IS CONTINUOUSENDENUMERATEENDEXERCISESSECTIONSETS OF COMPLETE ORTHOGONAL FUNCTIONSLABELSECCOFTHERE ARE SEVERAL SETS OF COMPLETE ORTHOGONAL FUNCTIONS THAT ARE USEDIN COMMON APPLICATIONS  WE WILL EXAMINE A FEW OF THE MORECOMMONLYUSED SETS  MOSTLY STATING RESULTS WITHOUT PROOFSSUBSECTIONTRIGONOMETRIC FUNCTIONSAS SEEN IN EXAMPLE REFEXMFS THE FAMILIAR TRIGONOMETRIC FUNCTIONSEMPLOYED IN FOURIER SERIES ARE ORTHOGONAL  THEY FORM A COMPLETE SETOF ORTHOGONAL FUNCTIONSINPUTFUNCTDIRORTHOGPOLYSUBSECTIONSINC FUNCTIONSINDEXSINC FUNCTIONTHE FUNCTION COMMONLY KNOWN AS A SINC FUNCTION SINCT  FRACSINPI TPI TCAN BE USED TO FORM A SET OF ORTHOGONAL FUNCTIONSBEGINEQUATION PKT  SINC2BTK2BLABELEQSINCSHIFTENDEQUATIONIT CAN BE SHOWN SEE EXERCISE REFEXSINCORTHOG FOR THE INNERPRODUCT LA FG RA INTINFTYINFTY FTGBARTDTTHAT LA PKTPLTRA  FRAC12BDELTAKL  IF FT ISA BANDLIMITED FUNCTION SUCH THAT ITS FOURIER TRANSFORM SATISFIES FOMEGA  0TEXT FOR  OMEGA NOT IN 2PI B2PI BTHEN IN THE SERIES REPRESENTATION FT  SUMK CK PKTTHE COEFFICIENTS ARE FOUND TO BEBEGINEQUATION CK  FRACLA FPKRALA PKPKRA  FK2BLABELEQSINCSAMPENDEQUATIONTHIS GIVES RISE TO THE FAMILIAR SAMPLING THEOREM REPRESENTATION OF ABANDLIMITED FUNCTION FT  SUMK FK2B FRACSIN2PI BTK2B2PI  BTK2BBEGINEXERCISES  ITEM PROVE PARSEVALS THEOREM FOR FOURIER TRANSFORMS IF Y1T    LEFTRIGHTARROW Y1OMEGA AND Y2T LEFTRIGHTARROW    Y2OMEGA THEN INTINFTYINFTY Y1T YBAR2TDT  FRAC12PIINTINFTYINFTY Y1OMEGA YBAR2OMEGADOMEGAITEM LABELEXSINCORTHOG SHOW FOR PKT DEFINED AS IN  REFEQSINCSHIFT THAT LA PK PLRA   FRAC12BDELTAKL    ALONG THE WAY SHOW THAT INTINFTYINFTY FRACSIN TT FRACSINTZTZ DT FRACPI SIN ZZHINT USE PARSEVALS THEOREM AND FOURIER TRANSFORMSITEM SHOW THAT REFEQSINCSAMP IS CORRECT FOR A BANDLIMITED  FUNCTION FTITEM SHOW THAT FZ  2BINTINFTYINFTY FT P0TZDTSO THAT FOR BANDLIMITED FUNCTIONS P0T BEHAVES LIKE A DELTAFUNCTIONENDEXERCISESINPUTLINALGDIRWAVELETSTEXINPUTLINALGDIRMATCHEDFTEXSETEXSECTREFSECGRADMINBEGINEXERCISESITEM LABELEXGRAMDET THERE IS A CONNECTION BETWEEN GRAMMIANS  INDEXGRAMMIAN AND LINEAR INDEPENDENCE INDEXLINEAR INDEPENDENCE  AS DEMONSTRATED IN THEOREM REFTHMGRAMMPD  WE EXPLORE THIS  CONNECTION FURTHER IN THIS PROBLEM  LET PBF1PBF2LDOTSPBFN BE A SET OF VECTORS AND LET  US SUPPOSE THAT THE FIRST K1 VECTORS OF THIS SET HAVE PASSED A  TEST FOR LINEAR INDEPENDENCE  WE FORM EBFK  CK1K PBF1  CK2K PBF2  CDOTS C1KPBFK1  PBFK AND WANT TO KNOW IF EBFK IS EQUAL TO ZERO FOR ANY SET OFCOEFFICIENTS CBFK  CK1K CK2KLDOTSC1K1TIF SO THEN PBFK IS LINEARLY DEPENDENT ON PBF1LDOTSPBFK1  LET AK  PBF1PBF2LDOTSPBFKBE A DATA MATRIX AND LET RK  AKHAK BE THE CORRESPONDINGGRAMMIANBEGINENUMERATEITEM SHOW THAT THE SQUARED ERROR CAN BE WRITTEN ASBEGINEQUATION EBFKH EBFK  SIGMAK2  CBFKHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFKLABELEQGRAMDET1ENDEQUATIONFOR SOME HBFK AND RKK  IDENTIFY HBFK AND RKKITEM DETERMINE THE MINIMUM VALUE OF SIGMAK2 BY MINIMIZING  REFEQGRAMDET1 WITH RESPECT TO CBFK SUBJECT TO THE  CONSTRAINT THAT THE LAST ELEMENT OF CBFK IS EQUAL TO 1  HINT  TAKE THE GRADIENT OF CBFKHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFK  LAMBDACBFKHDBF  1WHERE LAMBDA IS A LAGRANGE MULTIPLIER AND DBF 00LDOTS01T  SHOW THAT WE CAN WRITE THE CORRESPONDINGEQUATIONS ASBEGINEQUATIONBEGINBMATRIX RK1  HBFK  HBFKH  RKKENDBMATRIXCBFK  SIGMAK2 DBFLABELEQGRAMDET2ENDEQUATIONITEM SHOW THAT REFEQGRAMDET2 CAN BE MANIPULATED TO BECOME SIGMAK2  RKK  HBFKH RK11 HBFKTHE QUANTITY SIGMAK2 IS CALLED THE EM SCHUR COMPLEMENTINDEXSCHUR COMPLEMENT OFRK  IF SIGMAK20 THEN PBFK IS LINEARLY DEPENDENTENDENUMERATEEXSKIPSETEXSECTREFSECMINERR ITEM  SHOW THAT  REFEQXSTACKROW IS TRUE ITEM LABELEXREDUCEERR REFERRING TO REFEQREDUCERR SHOW THAT I  AAHA1AHIS POSITIVE SEMIDEFINITE AND HENCE THAT THE MINIMUM ERROREBFMIN HAS SMALLER NORM THAN THE ORIGINAL VECTOR XBF  HINTCONSIDER 0 LEQ  BXBF2 WHERE B  I  AAHA1AHEXSKIPSETEXSECTREFSECLINREG  ITEM CONSIDER THE SET OF DATA X   225359 QQUAD Y  42 5 2 1243BEGINENUMERATEITEM MAKE A PLOT OF THE DATAITEM DETERMINE THE BEST LEASTSQUARES LINE THAT FITS THIS DATA AND  PLOT THE LINEITEM ASSUMING THAT THE FIRST AND LAST POINTS ARE BELIEVED TO BE THE  MOST ACCURATE FORMULATE A WEIGHTING MATRIX AND COMPUTE A WEIGHTED  LEASTSQUARES LINE THAT FITS THE DATA  PLOT THIS LINEENDENUMERATEITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS2 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS3 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION Y APPROX CEAX AS A LINEAR  REGRESSION PROBLEM WITH REGRESSION PARAMETERS C AND AITEM FORMULATE THE REGRESSION Y APPROX AXB AS A LINEAR  REGRESSION PROBLEMITEM PERFORM THE COMPUTATIONS TO VERIFY THE SLOPE AND INTERCEPT OF  THE LINEAR REGRESSION IN REFEQLINREGRESSAITEM AS A MEASURE OF FIT IN A CORRELATION PROBLEM THE CORRELATION  COEFFICIENT ANALOGOUS TO REFEQCORRCOEFF CAN BE OBTAINED AS RHO  FRACLA XBF YBFRA  LA XBFONEBFRA LA YBFONEBFRA XBF  LA XBFONEBFRA YBF  LA YBFONEBFRATHE CORRELATION COEFFICIENT RHO  PM 1 IF X AND Y ARE EXACTLYFUNCTIONALLY RELATED AND RHO  0 IF THEY ARE INDEPENDENT  FOR THELINEAR REGRESSION IN REFEQ2REGRESS DETERMINE AN EXPLICITEXPRESSION FOR RHO ITEM GIVEN A SET OF DATA XBFI I12LDOTS M WHERE EACH VECTOR   XBFI IN RBBD FORMULATE THE LEASTSQUARES SOLUTION TO FIND   THE BEST DDIMENSIONAL PLANE FITTING THIS DATAITEM DEFINE AN INNER PRODUCT BETWEEN MATRICES X AND Y AS LA XY RA  TRACEXYHWHERE TRACECDOT IS THE SUM OF THE DIAGONAL ELEMENTS SEE SECTIONREFSECTPOSETRACE  WE WANT TO APPROXIMATE THE MATRIX Y BY THE SCALARLINEAR COMBINATION OF MATRICES X1X2LDOTSXM AS Y  C1 X1  C2 X2  CDOTS  CM XM  EUSING THE ORTHOGONALITY PRINCIPLE DETERMINE A SET OF NORMAL EQUATIONSTHAT CAN BE USED TO FIND C1C2LDOTSCM THAT MINIMIZE THEINDUCED NORM OF EITEM FOR THE ARMA INPUTOUTPUT RELATIONSHIP OF REFEQARMA  DETERMINE A SET OF LINEAR EQUATIONS FOR DETERMINING THE ARMA MODEL  PARAMETERS A1A2LDOTSAPB0B1LDOTSBQ ASSUMING  THAT THE MODEL ORDER PQ IS KNOWN AND THAT THE INPUT AND OUTPUT  ARE KNOWN  EXSKIP SETEXSECTREFSECLSFILT  ITEM VERIFY REFEQGRAMM3ITEM FOR THE DATA SEQUENCE 11235813  BEGINENUMERATE  ITEM WRITE DOWN THE DATA MATRIX A AND THE GRAMMIAN AH A USING    I THE COVARIANCE AND II THE AUTOCORRELATION METHODS  ITEM WE DESIRE TO USE THIS SEQUENCE TO TRAIN A SIMPLE LINEAR    PREDICTOR  THE DESIRED SIGNAL DT IS THE VALUE OF XT    AND THE DATA USED ARE THE TWO PRIOR SAMPLES  THAT IS  XT  A1 XT1  A2 XT2  ET WHERE ET IS THE PREDICTION ERRORDETERMINE THE LEASTSQUARES COEFFICIENTS FOR THE PREDICTOR USING THECOVARIANCE AND AUTOCORRELATION METHODSITEM DETERMINE THE MINIMUM LEASTSQUARES ERROR FOR BOTH METHODS  ENDENUMERATE ITEM BF COMPUTER EXERCISE GENERATE A SEQUENCE OF N RANDOM PM   1 BITS AND PASS THEM THROUGH A CHANNEL WITH TRANSFER FUNCTION  HZ  FRAC119Z1  THEN ADD NOISE WITH VARIANCE SIGMAN2  01  DETERMINE A LEASTSQUARES FILTER FOR THIS CHANNEL FOR VARIOUS VALUES OF THE DELAYEXSKIPSETEXSECTREFSECMMSSEFILT ITEM IMPLEMENTATION OF EQUALIZER IN EXAMPLE REFEXMEQ1 GENERATE   THE SIGNAL FT IN EXAMPLE REFEXMEQ1 BY CREATING AN AR1   SIGNAL WHICH IS PASSED THROUGH HCZ THEN CORRUPTED BY ADDITIVE   NOISE  THEN PASS THIS SIGNAL THROUGH THE EQUALIZER DESIGNED IN THAT   EXAMPLE  COMPARE THE EQUALIZED SIGNAL WITH THE SIGNAL DTITEM FOR A DATA SEQUENCE  XT THE CORRELATION MATRIX R IS R  BEGINBMATRIX 5  3  3  5 ENDBMATRIXAND THE CROSSCORRELATION VECTOR PBF WITH A DESIRED SIGNAL IS PBF  BEGINBMATRIX 2  5 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL WEIGHT VECTOR ITEM DETERMINE THE MINIMUM MEANSQUARED ERROR ENDENUMERATEITEM CONSIDER A ZEROMEAN RANDOM VECTOR XBF  X1X2X3 WITH  COVARIANCE COVXBF  EXBFXBFT  BEGINBMATRIX 1  7  5  7  4  2 5  2  3 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL COEFFICIENTS OF THE PREDICTOR OF X1 IN TERMSOF X2 AND X3 XHAT1  C1 X2  C2 X3ITEM DETERMINE THE MINIMUM MEANSQUARED ERRORITEM HOW IS THIS ESTIMATOR MODIFIED IF THE MEAN OF XBF IS E XBF   123TENDENUMERATEITEM CITEHAYKIN1996 A DISCRETETIME RADAR SIGNAL IS TRANSMITTED AS ST  A0 EJOMEGA0 TTHE SAMPLED NOISY RECEIVED SIGNALS ARE REPRESENTED AS XT  A1 EJOMEGA1 T  NUTWHERE OMEGA1 IS THE RECEIVED SIGNAL FREQUENCY IN GENERALDIFFERENT FROM OMEGA0 BECAUSE OF DOPPLER SHIFT AND NUT IS AWHITENOISE SIGNAL WITH VARIANCE SIGMAN2  LET XBFT  X0X1LDOTS XM1TBE A VECTOR OF RECEIVED SIGNAL SAMPLESBEGINENUMERATEITEM SHOW THAT R  EXBFT XBFHT  SIGMANU2 I  SIGMA12 SBFOMEGA1SBFHOMEGA1WHERE SBFOMEGA1  1EJOMEGA1EJ2OMEGA1LDOTSEM1  J OMEGA1TQQUAD TEXTAND QQUADSIGMA12  EA12ITEM THE TIME SERIES XT IS APPLIED TO AN FIR WIENER FILTER WITH  M COEFFICIENTS IN WHICH THE CROSSCORRELATION BETWEEN XT AND  THE DESIRED SIGNAL DT IS PRESET TO PBF  SBFOMEGA0DETERMINE AN EXPRESSION FOR THE TAPWEIGHT VECTOR OF THE WIENER FILTERENDENUMERATEITEM A CHANNEL WITH TRANSFER FUNCTION HCZ  FRAC112Z1AND OUTPUT UT IS DRIVEN BY AN AR1 SIGNAL DT GENERATED BY DT  4 DT1  NUTWHERE NUT IS A ZEROMEAN WHITENOISE SIGNAL WITH SIGMANU2 2  THE CHANNEL OUTPUT IS CORRUPTED BY NOISE NT WITH VARIANCESIGMAN2  15 TO PRODUCE THE SIGNAL FT  UT  NTDESIGN A SECONDORDER WIENER EQUALIZER TO MINIMIZE THE AVERAGE SQUAREDERROR BETWEEN FT AND DTITEM LABELEXLINPRED BF LINEAR PREDICTION INDEXLINEAR    PREDICTOR A COMMON APPLICATION OF WIENER FILTERING IS IN THE  CONTEXT OF LINEAR PREDICTION  LET DT  XT BE THE DESIRED  VALUE AND LET XHATT  SUMI1M WFI XTIBE THE PREDICTED VALUE OF XT USING AN MTH ORDER PREDICTOR BASEDUPON THE MEASUREMENTS XT1XT2LDOTSXTM ANDLET FMT  XT  XHATTBE THE EM FORWARD PREDICTION ERROR  INDEXFORWARD  PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORFORWARD THEN FMT  SUMI0M AFI XTIWHERE AF0  1 AND AFI  WFI I12LDOTSMASSUME THAT XT IS A ZEROMEAN RANDOM SEQUENCE  WE DESIRE TODETERMINE THE OPTIMAL SET OF COEFFICIENTS WFII12LDOTSMTO MINIMIZE EFMT2BEGINENUMERATEITEM USING THE ORTHOGONALITY PRINCIPLE WRITE DOWN THE NORMAL  EQUATIONS CORRESPONDING TO THIS MINIMIZATION PROBLEM  USE THE  NOTATION RJL  EXTL XBARTJ TO OBTAIN THE WIENERHOPF  EQUATION R WBFF  RBFWHERE R  EXBFT1XBFHT1 RBF  EXBFT1XT ANDXBFT1  XBART1XBART2LDOTSXBARTMTITEM DETERMINE AN EXPRESSION FOR THE MINIMUM MEANSQUARED ERROR PM   MIN EFMT2ITEM SHOW THAT THE EQUATIONS FOR THE OPTIMAL WEIGHTS AND THE MINIMUM  MEANSQUARED ERROR CAN BE COMBINED INTO EM AUGMENTED WIENERHOPF  INDEXWIENERHOPF   EQUATIONS AS BEGINBMATRIX R0  RBFH  RBF  R ENDBMATRIXBEGINBMATRIX 1  WBFF ENDBMATRIX  BEGINBMATRIXPM   ZEROBF ENDBMATRIXITEM SUPPOSE THAT XT HAPPENS TO BE AN ARM PROCESS DRIVEN BY WHITE NOISE  NUT SUCH THAT IT IS THE OUTPUT OF A SYSTEM WITH TRANSFER  FUNCTION HZ  FRAC11SUMK1M AK ZKSHOW THAT THE PREDICTION COEFFICIENTS ARE WFK  AK AND HENCETHE COEFFICIENTS OF THE PREDICTION ERROR FILTER FMT ARE AFI  AIHINT SEE SECTION REFSECARPROCESS WRITE DOWN THE YULEWALKEREQUATIONS INDEXYULEWALKER EQUATIONSHENCE CONCLUDE THAT IN THIS CASE THE FORWARD PREDICTION ERRORFMT IS A EM WHITENOISE SEQUENCE  THE PREDICTIONERRORFILTER CAN THUS BE VIEWED AS A EM WHITENING FILTER FOR THE SIGNALXT  INDEXWHITENING FILTERITEM NOW LET XHATTM  SUMI1M WBI XTI1BE THE EM BACKWARD PREDICTOR OF XTM USING THE DATA XTM1XTM2 LDOTS XT AND LET BMT  XTM  XHATTMBE THE BACKWARD PREDICTION ERROR  A BACKWARD PREDICTOR SEEMS STRANGE AFTER ALL WHY PREDICT WHAT WE SHOULD HAVE ALREADY SEEN  BUTTHE CONCEPT WILL HAVE USEFUL APPLICATIONS IN FAST ALGORITHMS FORINVERTING THE AUTOCORRELATION MATRIX  INDEXBACKWARD PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORBACKWARDSHOW THAT THE WIENERHOPF EQUATIONS FOR THE OPTIMAL BACKWARD PREDICTORCAN BE WRITTEN ASBEGINEQUATION R WBFB  OVERLINERBFBLABELEQBACKPRED1ENDEQUATIONWHERE RBFB IS THE BACKWARD ORDERING OF RBF  DEFINED ABOVEITEM FROM REFEQBACKPRED1 SHOW THAT RH OVERLINEWBFBB  RBFWHERE WBFBB IS THE BACKWARD ORDERING OF WBFB  HENCE CONCLUDETHAT OVERLINEWBFBB  WBFFTHAT IS THE OPTIMAL BACKWARD PREDICTION COEFFICIENTS ARE THE REVERSEDCONJUGATED OPTIMAL FORWARD PREDICTION COEFFICIENTSENDENUMERATEITEM LET XT  08 XT1   NUTWHERE NUT IS A REAL WHITENOISE ZEROMEAN UNITVARIANCE NOISEPROCESS  WE WANT TO DETERMINE AN OPTIMAL PREDICTORBEGINENUMERATEITEM IF THE ORDER OF THE PREDICTOR IS 2 DETERMINE THE OPTIMAL  PREDICTOR XHATTITEM IF THE ORDER OF THE PREDICTOR IS 1 DETERMINE THE OPTIMAL  PREDICTOR XHATTENDENUMERATEITEM BF RANDOM VECTORS THE MEANSQUARED METHODS TO THIS POINT HAVE  BEEN FOR RANDOM SCALARS  SUPPOSE WE HAVE THE RANDOM VECTOR  APPROXIMATION PROBLEM YBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFM  EBFIN WHICH WE DESIRE TO FIND AN APPROXIMATION YBF IN SUCH A WAY THATTHE NORM OF EBF IS MINIMIZED  LET US DEFINE AN INNER PRODUCTBETWEEN RANDOM VECTORS AS LA XBFYBF RA  TRACE EXBF YBFHBEGINENUMERATEITEM BASED UPON THIS INNER PRODUCT AND ITS INDUCED NORM DETERMINE A  SET OF NORMAL EQUATIONS FOR FINDING C1C2LDOTS CMITEM AS AN EXERCISE IN COMPUTING GRADIENTS USE THE FORMULA FOR THE  GRADIENT OF THE TRACE SEE APPENDIX REFAPPDXGRADIENT TO ARRIVE AT  THE SAME SET OF NORMAL EQUATIONSENDENUMERATEITEM BF MULTIPLE GAINSCALED VECTOR QUANTIZATION INDEXVECTOR QUANTIZATION LET X AND Y BE VECTOR SPACES OF THE SAME  DIMENSIONALITY  SUPPOSE THAT THERE ARE TWO SETS OF VECTORS XC1  XC2 SUBSET X  LET YC BE THE SET OF VECTORS EM POOLED FROM  XC1 AND XC2 BY THE INVERTIBLE MATRICES T1 AND T2  RESPECTIVELY  THAT IS IF XBF IN XCI THEN YBF  TI XBF  IS A VECTOR IN YC  INDICATE THAT A VECTOR YBF IN YC CAME  FROM A VECTOR IN XCI BY A SUPERSCRIPT I SO YBFI IN YC  MEANS THAT THERE IS A VECTOR XBF IN XC SUCH THAT YBFI  TI  XBF  DISTANCES RELATIVE TO A VECTOR YBFI IN YC ARE BASED  UPON THE L2 NORM OF THE VECTORS OBTAINED BY MAPPING BACK TO  XCI SO THATFOOTNOTETHE NOTATION DEFEQ MEANS IS DEFINED AS D2YBFIYBF  YBFI  YBF2 DEFEQ YBFI  YBFI TI1YBFI  YBF2  YBFI  YBFT WIYBFI  YBFWHERE WI  TITTI1  THIS IS A WEIGHTED NORM WITH THE WEIGHTINGDEPENDENT UPON THE VECTOR IN YC  NOTE IN THIS PROBLEM  CDOT1 AND  CDOT 2 REFER TO THE WEIGHTED NORM FOR EACH DATASET NOT THE L1 AND L2 NORMS RESPECTIVELY  WE DESIRE TO FIND A EM SINGLE VECTOR YBF0 IN Y THAT IS THEBEST REPRESENTATION OF THE DATA POOLED FROM BOTH DATA SETS IN THESENSE THAT SUMYBF IN YC YBF  YBF02  SUMYBF1 IN YC  YBF1  YBF012  SUMYBF2 IN YC YBF2  YBF022IS MINIMIZED  SHOW THAT YBF0  Z1 RBFWHERE Z  SUMYBF1 IN YC W1  SUMYBF2 IN YC W2 QQUADTEXTANDQQUADRBF  SUMYBF1 IN YC W1  YBF1  SUMYBF2 IN YC W2YBF2HINT THIS IS PROBABLY EASIER USING GRADIENTS THAN TRYING TO IDENTIFYTHE APPROPRIATE INNER PRODUCTEXSKIPSETEXSECTREFSECCMP  ITEM LABELEXBIASCORR LET X1X2LDOTSXN BE    SEQUENCE OF MEASURED DATAAN ESTIMATE OF THE CORRELATION FUNCTION OF THIS DATA ASSUMING THESEQUENCE IS ERGODIC IS INDEXAUTOCORRELATIONESTIMATE FROM DATA RHATK  FRAC1N SUMIK1N XI XBARIKBEGINENUMERATEITEM TAKE THE EXPECTATION ERHATK AND SHOW THAT IT IS NOT  EQUAL TO RK THE TRUE VALUE OF THE AUTOCORRELATIONITEM DETERMINE A SCALING FACTOR TO MAKE THE RHATK AN UNBIASED  ESTIMATEITEM WRITE A SC MATLAB FUNCTION THAT COMPUTES RHATK FROM  REFEQRHATNENDENUMERATESETEXSECTREFSECFREQFILTEXSKIPITEM LET XT YT AND VT BE CONTINUOUSTIME RANDOM PROCESSES  WITH YT  XT  VT AND SVS  1  DETERMINE AN OPTIMAL  CAUSAL FILTER HT  TO DETERMINE XT WHEN  BEGINENUMERATE  ITEM THE PSD OF XT IS SXS  FRACS2 16S4  53 S2  196ITEM THE PSD OF XT IS SXS  FRACS4  10 S2 9 S4  53 S2  196  ENDENUMERATEITEM SPECTRAL FACTORIZATION THE FEJERRIESZ THEOREM BECAUSE OFINDEXSPECTRAL FACTORIZATIONINDEXFEJERRIESZ THEOREMFEJERRIESZ THEOREMINDEXSQUARE ROOTOF A TRANSFER FUNCTION  THE IMPORTANCE OF THE CANONICAL FACTORIZATION IN SIGNAL PROCESSING  IT IS OF INTEREST TO DETERMINE WHEN A SQUARE ROOT OF A FUNCTION  EXISTS  IN THIS PROBLEM YOU WILL PROVE THE FOLLOWING IF WEJOMEGA   SUMNMM WN EJOMEGA N IS REAL AND WEJOMEGA GEQ 0 FOR  ALL OMEGA THEN THERE IS A FUNCTION YZ  SUMN0M YN ZNSUCH THAT WEJOMEGA  YEJOMEGA2BEGINENUMERATEITEM SHOW THAT WN  WBARNITEM SHOW THAT WBARZ  W1ZBARITEM SHOW THAT IF ZI IS A ROOT OF WZ THEN 1ZBARI IS A  ROOT OF WZITEM ARGUE THAT IF ZI  EJTHETAI IS A ROOT ON THE UNIT  CIRCLE THEN IT MUST HAVE EVEN MULTIPLICITY HINT USE THE FACT  THAT WEJOMEGAGEQ 0ITEM LET ZC  ZIMC  WZI  0 ZI LEQ 1 TEXT ONLY HALF    THE ROOTS ON Z1 BE THE SET OF ROOTS INSIDE AND HALF    THOSE ON THE UNIT CIRCLE  THEN ZC HAS M ELEMENTS AND WZ  A ZM PRODI1M ZZIPRODI1M ZZBARI  1FROM THIS FORM FIND YZENDENUMERATEEXSKIPSETEXSECTREFSECDTFFITEM LABELADDITIVEWHITEBF FILTERING IN WHITE NOISE LET XT YT AND VT BEDISCRETETIME RANDOM PROCESSES WITHYT  XT  VTAND BEGINALIGNEDSVZ    1 SXZ    FRACBZAZENDALIGNEDWHERE BZ AND AZ ARE POLYNOMIALS IN Z WITH THE DEGREE OFBZ STRICTLY BF LOWER THAN THE DEGREE OF AZ  FURTHERMOREASSUME RXVT EQUIV 0SHOW THAT REFADDITIVE HOLDS IN THE DISCRETETIME CASE THAT ISSHOW THAT THE OPTIMAL CAUSAL FILTER ISHZ  1  FRAC1SYZITEM LETYT  XT  VTWHERE BEGINALIGNEDRVT   FRAC23DELTAT RXT   FRAC1027LEFTFRAC12RIGHTTENDALIGNEDWITH EXT  EVT  EXTVT  0SHOW THATBEGINENUMERATEITEM SYZ  FRAC1FRACZ31FRAC13Z1ZOVER 211OVER 2ZAND THUS OBTAIN SYZ AND SYZITEM LEFTFRACSXYZSYZRIGHT     FRACFRAC131  FRAC12Z AND THUS THAT THE WIENER  FILTER IS HZ  FRACFRAC131  FRAC13Z ITEM CONFIRM THAT THIS RESULT AGREES WITH THE RESULTS OFEXERCISE REFADDITIVEWHITEENDENUMERATEITEM LET XT YT AND VT BE DISCRETETIME RANDOM PROCESSES  WITH YT  XT  VT SVZ  1 AND SXZ  FRACZ4  90067 Z3  2804Z2 90067Z 1Z4 20111Z3  30446Z2 20111Z1DETERMINE THE FILTER HZ TO OPTIMALLY PREDICT XT2EXSKIPSETEXSECTREFSECDUALAPPROXITEM LABELEXINTSYSRESP LET HT BE THE IMPULSE RESPONSE OF A  SYSTEM AND LET YT  XTHT  SHOW THAT INT0T YTDT  XTKTWHERE KT IS THE INTEGRAL OF THE IMPULSE RESPONSE KT  INT0T HSDSITEM A SYSTEM IS KNOWN TO HAVE IMPULSE RESPONSE HT  3E2T   4E5T AND IS INITIALLY RELAXED INITIAL CONDITIONS ARE ZERO  DETERMINE AN INPUT XT SO THAT THE OUTPUT SATISFIES THE  CONDITIONS Y2  2 QQUAD TEXTANDQQUAD INT02 YTDT  3IN SUCH A WAY THAT THE INPUT ENERGY XT2  INT02XT2DT IS MINIMIZED  ITEM LET HT  02T  304T FOR T GEQ 0 BE THE IMPULSE    RESPONSE OF A DISCRETETIME SYSTEM WITH ZERO INITIAL CONDITIONS    IT IS DESIRED TO DETERMINE A CAUSAL INPUT SEQUENCE XT SUCH    THAT THE OUTPUT YT  HTXT SATISFIES THE CONSTRAINTS BEGINALIGNEDY10  5 SUMJ010 YJ  2ENDALIGNEDAND SUCH THAT THE INPUT ENERGY SUMK010 XT2 ISMINIMIZED  FORMULATE THIS AS A DUAL APPROXIMATION PROBLEM AND FINDTHE MINIMIZING SEQUENCE XT  EXSKIP SETEXSECTREFSECLS2ITEM CITELUENBERGER1969  USING THE PROJECTION THEOREM SOLVE THE  FINITE DIMENSIONAL PROBLEM BEGINALIGNEDTEXTMINIMIZE   XBFH Q XBF TEXTSUBJECT TO   A XBF  BBFENDALIGNEDWHERE XBF IN CBBN Q IS A POSITIVE DEFINITE SYMMETRIC MATRIXAND A IS AN MATSIZEMN MATRIX WITH M  NITEM CITELUENBERGER1969 LET XBF BE A VECTOR IN A HILBERT SPACE  S AND LET  XBF1 XBF2LDOTSXBFN AND   YBF1YBF2LDOTS YBFM BE SETS OF LINEARLY INDEPENDENT  VECTORS IN S  WE DESIRE TO MINIMIZE  XBF  XBFHAT WHERE  THE NORM IS THE INDUCED NORM WHILE SATISFYING XBFHAT IN M  LSPAN XBF1XBF2LDOTS XBFMAND LA XBFHAT YBFIRA  CI I12LDOTS MBEGINENUMERATEITEM FIND EQUATIONS FOR THE SOLUTION WHICH ARE SIMILAR TO THE NORMAL  EQUATIONS  ITEM GIVE A GEOMETRIC INTERPRETATION OF THE EQUATIONS ENDENUMERATE ITEM CITELUENBERGER1969 CONSIDER THE PROBLEM OF FINDING THE VECTOR   XBF OF MINIMUM NORM SATISFYING LA XBFYBFIRA GEQ CI FOR   I12LDOTS N WHERE THE YBFI ARE LINEARLY INDEPENDENT   BEGINENUMERATE   ITEM SHOW THAT THIS PROBLEM HAS A UNIQUE SOLUTION   ITEM SHOW THAT A NECESSARY AND SUFFICIENT CONDITION THAT  XBF  SUMI1N AI YBFI  IS A SOLUTION IS THAT THE VECTOR ABF WITH COMPONENTS AI SATISFY  RT ABF GEQ CBF QQUAD TEXTANDQQUAD ABF GEQ ZEROBF  AND THAT AI  0 IF LA XBFYBFIRA  CI  R IS THE GRAMMIAN MATRIX OF YBF1YBF2LDOTS YBFN   ENDENUMERATEEXSKIPSETEXSECTREFSECGFSITEM LABELEXORTHOGDFT SHOW THAT THE FUNCTIONS DEFINED BY PKT FRAC1SQRTN EJ2PI KTNARE ORTHONORMAL WITH RESPECT TO THE INNER PRODUCT LA XTYTRA  SUMK0N1 XTYBARTITEM LET GT  ET2 FOR 0 LEQ T LEQ PI AND LET FT BE  THE PIPERIODIC EXTENSION OF GT FT  SUMK GTKPIBEGINENUMERATEITEM FIND THE FOURIER SERIES COEFFICIENTS OF FTITEM FIND THE SUM OF THE SERIES SUMN LEFTFRAC22PI2 FRAC11 16N2RIGHTHINT USE PARSEVALS THEOREMENDENUMERATEEXSKIPSETEXSECTREFSECCOFITEM PROPERTIES OF THE BERNSTEIN POLYNOMIALS AND RELATED FORMULAS  PROVE THE PROPERTIES REFEQBPROP1 REFEQBPROP2 AND  REFEQBPROP3   HINT USING THE BINOMIAL THEOREM SUMJ0N N CHOOSE J XJ YNJ  XYNSHOW THAT  SUMJ0N JNN CHOOSE J XJ YNJ     XXYN1AND SUMJ0N JN2 N CHOOSE J XJ YNJ  11N    X2 XYN2  XNXYN1 ITEM SHOW THAT THE LEGENDRE POLYNOMIAL PNT IS A SOLUTION TO THE   DIFFERENTIAL EQUATION  1T2Y  NN1Y  0   ITEM LABELEXCHEBPOLY1 SHOW THAT THE ORTHOGONALITY RELATION FOR     CHEBYSHEV POLYNOMIALS IN REFEQCHEBORTHOG IS TRUEITEM USING REFEQCHEB1 DETERMINE T2T AND T3TITEM SHOW THAT THE DEFINITION OF CHEBYSHEV POLYNOMIALS  REFEQCHEB1 SATISFIES THE RECURRENCE IN REFEQCHEBRECURR  FOR T 1 SHOW FOR T1 THAT TNT  COSHNCOSH1T SATISFIES  THE RECURSION REFEQCHEBRECURRITEM BF THE CHRISTOFFELDARBOUX FORMULA  BEGINENUMERATE  ITEM USING REFEQPOLYRECURR SHOW THAT THE POLYNOMIALS    PKT ORTHOGONAL WITH RESPECT TO THE INNER PRODUCT LA FGRAW     INTAB FTGTDT SATISFY INTAB PNT PN1TWTDT  ANALSO SHOW THAT CN  AN1ITEM CONSIDER THE PARTIAL SUM SNT  SUMK0N LA FPKRAW PKTSHOW THAT THE SUM CAN BE WRITTEN AS SNT INTAB FY KNXY WYDYWHERE KNXY  FRACANPN1XPNY  PNXPN1YXYAND WHERE AN COMES FROM REFEQPOLYRECURR  WALTER P 78THIS FORMULA FOR KNXY IS KNOWN AS THE EM CHRISTOFFELDARBOUXFORMULA AND IS ANALOGOUS TO THE DIRICHLET KERNEL OF FOURIERSERIES  HINT FORM XYKXY AND USE THE RESULTS FROM PART AINDEXDIRICHLET KERNEL INDEXCHRISTOFFELDARBOUX FORMULA SEE SECTION REFSECDIRICHLETKERNEL  ENDENUMERATEITEM IT IS ALSO POSSIBLE TO DEFINE ORTHOGONAL POLYNOMIALS OF A  DISCRETE VARIABLE USING THE INNER PRODUCT LA FGRA  SUMI WXI FXIGXIWHERE THE XI ARE INTEGERS IN THE INTERVAL A LEQ XI LEQ B AND WXI 0  THIS AMOUNTS TO DEFINING THE INNER PRODUCT USINGDELTA FUNCTIONS IN THE INTEGRAL  ITEM SHOW THAT THE EACH OF THE POLYNOMIALS PRODUCED BY  ORTHOGONALIZING 1TT2LDOTS USING THE GRAMSCHMIDT  PROCEDURE OVER THE INTERVAL AB HAS ZEROS WHICH ARE REAL  SIMPLE AND LOCATED IN AB ITEM RECURRENCE FOR ORTHOGONAL POLYNOMIALS   BEGINENUMERATE   ITEM SHOW THAT THE LEGENDRE POLYNOMIALS SATISFY THE RECURSION  PN1T  FRAC2N1N1 T PNT  FRACNN1 PN1T  USE THIS RECURRENCE TO COMPUTE P3T P4T AND P5T ITEM THE CHEBYSHEV POLYNOMIALS SATISFY THE RECURSION   TN1T  2T TNT  TN1  USE THIS RECURRENCE TO FIND T3T T4T AND T5T   ENDENUMERATEITEM IN THIS EXERCISE WE INTRODUCE THE IDEA OF EM GAUSSIAN    QUADRATURE INDEXGAUSSIAN QUADRATURE A FAST AND IMPORTANT  METHOD OF NUMERICAL INTEGRATION  INDEXNUMERICAL INTEGRATIONSEEGAUSSIAN QUADRATURE THE IDEA IS TO APPROXIMATE THE  INTEGRAL AS A SUMMATION INTAB FT DT APPROX SUMI1M AI FTIUNLIKE MANY CONVENTIONAL NUMERICAL INTEGRATION FORMULAS IN GAUSSIANQUADRATURE THE ABSCISSAS ARE NOT EVENLY SPACED  THE PROBLEM IS TOFIND THE TI ABSCISSAS AND AI WEIGHTS SO THATTHE INTEGRAL IS AS ACCURATE AS POSSIBLE  IN THE GAUSSIAN QUADRATUREMETHOD OF NUMERIC INTEGRATION FOR POLYNOMIALS UP TO DEGREE 2M1THE RESULT OF THE INTEGRATION IS EM EXACT  FOR SUFFICIENTLY SMOOTHNONPOLYNOMIAL FUNCTIONS THE METHOD IS OFTEN VERY ACCURATE  THESOLUTION MAKES SIGNIFICANT USE OF ORTHOGONAL POLYNOMIALS   FORPURPOSES OF THIS EXERCISE WE WILL ASSUME THE INNER PRODUCT LAFGRA  INT11 FTGTDTBEGINENUMERATEITEM  AS THIS FIRST PART SHOWS WITHOUT  LOSS OF GENERALITY WE MAY  RESTRICT ATTENTION TO THE INTERVAL A1 B1SHOW THAT FOR THE INTEGRAL INTAB GXDXTHE SUBSTITUTION  T  FRAC1BA2X  ABLEADS TO AN INTEGRAL OF THE FORM INT11 FTDTHENCE THE LIMITS OF A AND B CAN BE CONVERTED TO LIMITS OF 1 TO 1ITEM IF PNT IS A SET OF POLYNOMIALS ORTHOGONAL OVER  11 WHERE PNT IS A POLYNOMIAL OF DEGREE N SHOW THAT LA PTPMTRA  0FOR ALL POLYNOMIALS PT OF DEGREE LEQ M1ITEM LET FT BE A POLYNOMIAL OF DEGREE 2M1  SHOW THAT FT  CAN BE WRITTEN AS FT  QTPNT  RTWHERE QT AND RT ARE OF DEGREE LEQ M1  HINT DIVIDEITEM SHOW THAT THERE ARE SERIES EXPANSIONS QT  SUMK0M1 ALPHAK PKT QQUADTEXTAND QQUAD   RT  SUMK0M1 BETAK PKTITEM SHOW THATBEGINEQUATION INT11 FTDT  BETA0 INT11 P0TDTLABELEQGINT3ENDEQUATIONITEM LET T1 T2LDOTS TM BE THE ROOTS OF PMT  SHOW THATBEGINEQUATION SUMI1M AI FTI  SUMK0M1 BETAK SUMI1M AIPKTILABELEQGINT4ENDEQUATION BEGINALIGNED   SUMI1M AI FTI  SUMI1M AI QTIPMTI  RTI   SUMI1M AI RTI  SUMI1M AI SUMK0M1 BETAK PKTI   SUMK0M1 BETAK SUMI1M AI PKTI ENDALIGNED ITEM SHOW THAT IF THE WEIGHTS AI ARE CHOSEN SO THAT SUMI1M AI PKTI  BEGINCASES  INT11 P0TDT  K0   0  K12LDOTS N1ENDCASESTHEN REFEQGINT4 CAN BE WRITTEN ASBEGINEQUATION SUMI1M AI FTI  B0 INT11 P0TDTLABELEQGINT5ENDEQUATIONITEM WRITE REFEQGINT5 AS A MATRIX EQUATION FOR THE WEIGHTS  AIITEM HENCE EQUATING REFEQGINT3 AND REFEQGINT5 WRITE  DOWN THE FORMULA FOR GAUSSIAN QUADRATUREITEM GENERALIZE THIS TO FINDING INT11 WTFTDT WHERE THE  POLYNOMIALS PKT ARE ORTHOGONAL WITH RESPECT TO THE INNER  PRODUCT LA FGRA  INT11 FTGTWTDTENDENUMERATE  ITEM PROVE PARSEVALS THEOREM FOR FOURIER TRANSFORMS IF Y1T    LEFTRIGHTARROW Y1OMEGA AND Y2T LEFTRIGHTARROW    Y2OMEGA THEN INTINFTYINFTY Y1T YBAR2TDT  FRAC12PIINTINFTYINFTY Y1OMEGA YBAR2OMEGADOMEGAITEM LABELEXSINCORTHOG SAMPLING THEOREM REPRESENTATIONS  BEGINENUMERATE  ITEM SHOW FOR PKT DEFINED AS IN  REFEQSINCSHIFT THAT LA PK PLRA   FRAC12BDELTAKL    ALONG THE WAY SHOW THAT INTINFTYINFTY FRACSIN TT FRACSINTZTZ DT FRACPI SIN ZZHINT USE PARSEVALS THEOREM AND FOURIER TRANSFORMSITEM SHOW THAT REFEQSINCSAMP IS CORRECT  FOR A BANDLIMITED FUNCTION FTITEM SHOW THAT IF FT IS BANDLIMITED TO B HZ  FZ  2BINTINFTYINFTY FT P0TZDTTHUS FOR BANDLIMITED FUNCTIONS P0T BEHAVES LIKE A DELTAFUNCTION  ENDENUMERATEEXSKIPITEM SHOW THAT IF PHIT IS NORMALIZED THEN 2J2PHI2J  T IS NORMALIZEDITEM IN REFEQTWOSCALE2 SHOW THAT THE COEFFICIENTS CN MUST  SATISFY SUMN CN  2 ITEM USING REFEQWAVEORTHOG1 SHOW THAT   BEGINENUMERATE   ITEM THE SET OF FUNCTIONS   2J2PHI2J T  N N IN ZBB  FORMS AN ORTHOGONAL SET FOR EACH   FIXED J ITEM THE SET OF FUNCTIONS  2J2 PSI2JT  N N IN ZBB FORMS AN   ORTHOGONAL SET FOR EACH FIXED J     ENDENUMERATEITEM SHOW THAT THERE IS NO ORTHOGONAL SCALING FUNCTION DEFINED BY A  TWOSCALE EQUATION REFEQTWOSCALE2 WITH EXACTLY THREE NONZERO  COEFFICIENTS C0 C1 AND C2  ITEM FOR THE MULTIRESOLUTION ANALYSIS    BEGINENUMERATE    ITEM SHOW THAT WJ PERP WJ    ITEM SHOW THAT FOR J  J VJ   VJ OPLUS BIGOPLUSK0JJ1 WJK    ENDENUMERATE  ITEM SHOW THAT IF PHIT OBEYS THE TWOSCALE RELATIONSHIP IN    REFEQTWOSCALE1 AND IF PHIHATOMEGA REPRESENTS THE    FOURIER TRANSFORM OF PHIT THEN PHIHATOMEGA  M0OMEGA2PHIHATOMEGA2WHERE BEGINEQUATIONM0OMEGA  FRAC1SQRT2 SUMN HN EJNOMEGALABELEQM0ENDEQUATIONIS THE SCALED DISCRETETIME FOURIER TRANSFORM OF THE COEFFICIENTSEQUENCEITEM BF DECIMATION INDEXDECIMATION INDEXMULTIRATE PROCESSINGBECAUSE OF THE CONNECTION OF  WAVELET TRANSFORMS WITH MULTIRATE SIGNALING IT IS WORTHWHILE TO  EXAMINE THE TRANSFORM OF DECIMATED SIGNALS  YOU WILL SHOW THAT IF  YN IS A DECIMATION OF XN YN  XNDTHENBEGINEQUATIONYZ  FRAC1D SUMK0D1 XEJ2PI KDZ1DLABELEQDECIMATEENDEQUATIONBEGINENUMERATEITEM LET PN BE THE PERIODIC SAMPLING SEQUENCE PN  BEGINCASES  1  N  0 PM D PM 2D LDOTS   0  TEXTOTHERWISEENDCASESSHOW THAT PN  FRAC1DSUMK0D1 EJ2PI KNDITEM LET ZN  XNPN  THEN YN  ZND  SHOW THAT YZ  SUMM YM ZM  SUMM ZMITEM FINALLY SHOW THAT REFEQDECIMATE IS TRUEENDENUMERATEITEM SHOW THAT THE ORTHOGONALITY CONDITION REFEQWAVEORTHOG1  IS EQUIVALENT TO  M0OMEGA22  M0OMEGA2PI2  1HINT RECOGNIZE THAT REFEQWAVEORTHOG1 IS A DECIMATEDCONVOLUTION AND USE THE FACT THAT IF THE FOURIER TRANSFORM OF ASEQUENCE ZN IS ZEJOMEGA THEN THE FOURIER TRANSFORM OF Z2NIS FRAC12ZEJOMEGA2  ZEJOMEGA2PI ITEM COMPUTER EXERCISE  IN THIS EXAMPLE YOU WILL BE INTRODUCED TO A   RUDIMENTARY APPROACH TO DATA COMPRESSION USING WAVELETS  WRITE A   PROGRAM WHICH WAVELET TRANSFORMS DATA THEN TRUNCATES THE DATA USING   A PRESET THRESHOLD THEN INVERSE TRANSFORMS THE DATA  USING   SAMPLED SPEECH OR MUSIC DATA EXPLORE THE QUALITY OF THE   INVERSETRANSFORMED DATA AS A FUNCTION OF THE THRESHOLD  DETERMINE   HOW MANY COEFFICIENTS ARE SET TO ZERO AS A FUNCTION OF THE THRESHOLDEXSKIPITEM LABELEXMF1 LET PHIT BE A ONEDIMENSIONAL BASIS FUNCTION  FOR DIGITAL TRANSMISSION OF THE FORM PHIT  UT  UTTA UNIT PULSE  ASSUME THAT ST  PHIT IS TRANSMITTED  LETRT  ST NOISEFREE RECEPTION  SHOW THE OUTPUT OF THECORRELATOR Y1T  INT0T RUPHIUDUAND THE OUTPUT OF THE MATCHED FILTER WITH IMPULSE RESPONSE HT PHITT  Y2T  RTHTSHOW THAT AT THE SAMPLE INSTANT T  T Y1T  Y2TITEM FOR THE BASIS SIGNALS SHOWN IN FIGURE REFFIGMFEX2 DRAW A  SIGNAL CONSTELLATION  SUCH A SIGNALING TECHNIQUE IS CALLED EM    PULSEPOSITION MODULATIONITEM LET  PHIMT  BEGINCASESCOS2PI FC   2PI M DELTA FT  0 LEQ T LEQ T 0  TEXTOTHERWISEENDCASESFOR M01LDOTSM1 BE A SET OF BASIS FUNCTIONS  DETERMINE THE  MINIMUM FREQUENCY SEPARATION DELTA F SUCH THAT INT0T PHIMT PHIKTDT  0FOR K NEQ M  ASSUME THAT FC T  N FOR SOME INTEGER NDIGITAL TRANSMISSION WITH SUCH SIGNALS IS CALLED FREQUENCYSHIFTKEYING INDEXFREQUENCYSHIFT KEYINGITEM SPREADSPECTRUM MULTIPLE ACCESS  IN THIS EXERCISE WE EXAMINE  MATCHED FILTERS FOR A MORE COMPLICATED SCENARIO SPREAD SPECTRUM  MULTIPLE ACCESS INDEXSPREAD SPECTRUM MULTIPLE ACCESS  IN THIS MODEL  K USERS ARE TRANSMITTING  SIMULTANEOUSLY WITH THE KTH USER TRANSMITTING A SIGNAL SKT  SUMN BKN SQRT2 WK PHIKTNTWHERE PHIKT IS THE KTH USERS UNIQUE WAVEFORM A SIGNAL WITHSUPPORT OVER 0T  THE RECEIVED SIGNAL CONSISTS OF THE SUM OF EACHUSERS DELAYED SIGNAL APPEARING IN ADDITIVE NOISE RT  SUMK1K SUMN BKN WK PHIKTNT TAUK  ZTTHE USERS BASIS FUNCTIONS ARE EM NOT NECESSARILY ORTHOGONALASSUME THAT THE USERS ARE ORDERED SO THAT TAU1 LEQ TAU2 LEQCDOTS LEQ TAUK  T  A MATCHEDFILTER OR CORRELATOR OUTPUT ISOBTAINED FOR EACH USER OVER THE NTH BIT INTERVAL AS YKN  INTINFTYINFTY RT PHIKTNT TAUKLET YBFN  Y1NY2NLDOTSYKNT BE THE VECTOR OFMATCHED FILTER OUTPUTS FOR ALL USERS AT INTERVAL N  BEGINENUMERATEITEM SHOW THAT YBFN  H1BN1  H0BN  H1BN1WBF  ZBFNWHERE HM IS A CORRELATION MATRIX WITH ELEMENTS HIJM  INTINFTYINFTY PHIITTAUIPHIJTMTTAUJDTB IS A DIAGONAL MATRIX OF BITS BN  DIAGB1NB2NLDOTSBKN WBF  W1W2LDOTSWKT AND ZBFN Z1NZ2NCDOTS ZKNT WHERE ZKN  INT ZT PHIKT  NT  TAUK DTITEM IF ZT IS WHITE WITH EZTZTS  SIGMAZ2 DELTATS  SHOW THAT ZBFN SATISFIES EZBFN ZBFTM  BEGINCASES  SIGMAZ2 H0  NM   SIGMAZ2 H1  N  M1   SIGMAZ2 H1  N  M1   0  TEXTOTHERWISEENDCASESENDENUMERATEENDEXERCISESSECTIONREFERENCESTHE HILBERT APPROXIMATION THEORY PRESENTED HERE IS SUMMARIZED FROMCITELUENBERGER1969 AND CITEKEENER  SOME OF THE DISCUSSION ABOUTTHE GRAMMIAN MATRIX WAS DRAWN FROM CITESCHARFL1991THE VARIOUS WINDOWING METHODS ARE DESCRIBED IN CITECHAPTER11HAYKIN1996  A DISCUSSION OF LEASTSQUARES ANDMINIMUM MEANSQUARES FILTERING IS INCITEHAYKIN1996PROAKISRADERSCHARFL1991 OUR DISCUSSION OF WIENER FILTERING IS DRAWN FROMCITEKAILATHFILTBOOK AND CITESOLODOVNIKOV  A THOROUGH DISCUSSIONOF THE SPECTRAL FACTORIZATION PROBLEM APPEARS IN CITEPAPOULIS1977THE GRAMSCHMIDT PROCEDURE IS DISCUSSED IN MOST BOOKS ON LINEARALGEBRA  SPECIFIC RESULTS ON NUMERIC ACCURACY OF THE METHOD CAN BEFOUND IN CITEGVLSEVERAL VARIANTS ON LEASTSQUARES AND CONSTRAINED LEASTSQUARESINCLUDING PSEUDOCODE FOR SEVERAL USEFUL ALGORITHMS ARE INCITELAWSONHANSENORTHOGONAL FUNCTIONS ARE WIDELY DISCUSSED IN CITEABRAMOWITZINCLUDING AN EXTENSIVE TABLE OF POLYNOMIALS ORTHOGONAL WITH RESPECT TOMANY WEIGHTING FUNCTIONS AND THEIR PROPERTIES  IN ADDITION TOORTHOGONAL POLYNOMIALS IN CONTINUOUS TIME THERE ARE ALSO ORTHOGONALPOLYNOMIALS IN DISCRETE VARIABLES  THESE ARE SUMMARIZED INCITEABRAMOWITZ AND EXAMINED MORE THOROUGHLY INCITEERDELYI1953 AND CITESZEGO1967  A RECENT BOOK DESCRIBING AVARIETY OF ORTHOGONAL FUNCTIONS AND THEIR SMOOTHNESS PROPERTIES ISCITEWALTER1994THE USE OF THE FUNCTION SINXX THE SINC FUNCTION AS ANORTHOGONAL BASIS IS INTRODUCED IN CITEKEENER  AN EXTENSIVEDISCUSSION OCCURS IN CITESTENGERBOOK AND CITESTENGERPAPERTHERE HAS BEEN AN EXPLOSION OF LITERATURE ON WAVELETS AND WAVELETTRANSFORMS  THE DEFINITIVE REFERENCE IS PROBABLYCITEDAUBECHIES1992 SEE ALSO CITEDAUBECHIES3  OTHER BOOKS WITHBROAD COVERAGE CITECHUI1992MALLAT1998  AMONG THE GENERALIZATIONSDISCUSSED IN THESE BOOKS ARE BIORTHOGONAL WAVELETS IN WHICH DIFFERENTFILTERS ARE USED TO RECONSTRUCT THE SIGNAL THAN TO ANALYZE ITWAVELET PACKETS CHOOSING DIFFERENT TREES OF COEFFICIENTS ANDSEVERAL OTHER FAMILIES OF WAVELETS  A RECENT TUTORIAL ISCITEBURRUSGOPINATH  A THOROUGH DISCUSSION OF IMPLEMENTATION OFWAVELET TRANSFORMS AND A VARIETY OF OTHER USEFUL TRANSFORMS AS WELLIS PROVIDED IN CITEWICKERHAUSER1994  A DEFINITIVE REFERENCE ONMULTIRATE SIGNAL PROCESSING IS CITEVAIDYANATHAN1993 FOR A SOLIDINTRODUCTION TO THIS AREA SEE CITEVAIDYANATHAN1990IRLS IS DISCUSSED IN CITEBURRUS1994 AND REFERENCES THEREIN WHERETHE NUMBER OF ITERATIONS REQUIRED TO DESIGN A FILTER IS CLOSELYEXAMINED  AN ALTERNATIVE VIEWPOINT ON ESTIMATION USING THE L1NORM INDEXL1NORM ESTIMATIONL1NORM ESTIMATION FOR SPECTRALESTIMATION IS INVESTIGATED INCITESCHROEDER1989SCHROEDER1990DENOEL1985  WARD1984  A MORE THOROUGH TREATMENT IS PRESENTED INCITEBLOOMFIELD1983THE VECTOR SPACE VIEWPOINT SIGNAL CONSTELLATIONS AND MATCHED FILTERSAND ARE PRESENTED IN EVERY TEXT ON DIGITAL COMMUNICATIONS  SEE FOREXAMPLE CITEWOZENCRAFT CITEPROAKIS3RDED OR CITEBLAHUTCOMMA HISTORICAL TREATMENT OF ORTHOGONAL FUNCTIONS USED IN SIGNALING ISGIVEN IN CITEHARMUTH WHICH ALSO PRESENTS OTHER USEFUL ORTHOGONALFUNCTIONS OTHER THAN THOSE PRESENTED HERETHERE IS A TREMENDOUS LITERATURE ON ORTHOGONAL POLYNOMIALS  A RECENTSURVEY IS CITEWALTER1994  A CLASSIC REFERENCE IS CITESZEGO1967ADDITIONAL INFORMATION IS FOUND IN CITEABRAMOWITZ LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERSOME IMPORTANT MATRIX FACTORIZATIONSLABELCHAPMATFACTTHERE ARE SOME MATRIX FACTORIZATIONS THAT ARISE COMMONLY ENOUGH INMATRIX ANALYSIS IN GENERAL AND IN SIGNAL PROCESSING IN PARTICULARTHAT THEY WARRANT SOME SPECIFIC ATTENTION  IN THIS CHAPTERFACTORIZATIONS ARE DISCUSSED WHICH FORM THE HEART OF MANY SIGNALPROCESSING ROUTINES  THE FACTORIZATIONS PRESENTED IN THIS CHAPTER AREAS FOLLOWSBEGINDESCRIPTIONITEMLU A SQUARE MATRIX A CAN BE FACTORED AS A  LU WHERE L  IS A LOWER TRIANGULAR MATRIX WITH ONES ON THE MAIN DIAGONAL AND U  IS UPPER TRIANGULAR  ITS MAIN APPLICATION IS IN THE NUMERICAL  SOLUTION OF THE PROBLEM AXBF  BBF INDEXMATRIX FACTORIZATIONSLUINDEXLU FACTORIZATIONINDEXMATRIX FACTORIZATIONSSEESVDITEMCHOLESKY A HERMITIAN SYMMETRIC POSITIVE DEFINITE MATRIX A  CAN BE FACTORED AS   A  LLH  INDEXSQUARE ROOTOF A MATRIX  WHERE L IS LOWER TRIANGULAR  THE CHOLESKY FACTORS OF A MATRIX MAY  BE REGARDED AS THE SQUARE ROOT OF THE MATRIX A  CLOSELY  RELATED IS THE FACTORIZATION A  LDLH WHERE D IS DIAGONAL OR  A  UDUH WHERE UU IS UPPER TRIANGULAR  THE CHOLESKY  FACTORIZATION IS USED IN SIMULATION TO COMPUTE A VECTOR NOISE OF  DESIRED COVARIANCE AND IN SOME ESTIMATION AND KALMAN FILTERING  ROUTINESINDEXMATRIX FACTORIZATIONSCHOLESKYINDEXCHOLESKY FACTORIZATIONITEMQR A GENERAL MATRIX A CAN BE FACTORED AS   A  QR    WHERE Q IS A UNITARY MATRIX QQH  I AND R IS UPPER  TRIANGULAR  THE QR FACTORIZATION IS USED IN THE SOLUTION OF  LEASTSQUARES PROBLEMSINDEXMATRIX FACTORIZATIONSQRINDEXQR FACTORIZATIONENDDESCRIPTIONA FACTORIZATION IMPORTANT ENOUGH TO WARRANT ITS OWN CHAPTER IS THESINGULAR VALUE DECOMPOSITION SVD IN WHICH A IS FACTORED AS A  USIGMA VHWHERE U AND V ARE UNITARY AND SIGMA IS DIAGONAL  THE SVD ANDITS APPLICATIONS IS PRESENTED IN CHAPTER REFCHAPSVD INDEXMATRIX FACTORIZATIONSSVD INDEXSVDINPUTLINALGDIRLUFACTSECTIONTHE CHOLESKY FACTORIZATIONLABELSECCHOLESKYINDEXMATRIX FACTORIZATIONSCHOLESKY INDEXCHOLESKY FACTORIZATIONTHE CHOLESKY FACTORIZATION IS USED TO COMPUTE A SQUARE ROOTINDEXMATRIX SQUARE ROOT OF A POSITIVEDEFINITE MATSIZEMMHERMITIAN MATRIX AS B  LLHWHERE L IS LOWER TRIANGULAR  OCCASIONALLY THE L MATRIX ISNORMALIZED TO PRODUCE A MATRIX LTILDE THAT IT HAS ONES ALONG THEMAIN DIAGONAL AND THE SCALING FACTOR IS INCORPORATED IN A DIAGONALMATRIX FACTOR AS LH  BEGINBMATRIXL11   L22   DDOTS  LMMENDBMATRIXLTILDEH  SQRTD UTHEN WE CAN WRITE B  UHD UWHERE D  DIAGL112 L222 LDOTS LMM2 BEGINEXAMPLE FOR THE B SHOWN WE HAVEBEGINALIGNEDB  BEGINBMATRIX 4  8  12 8 20   20 12  20  41 ENDBMATRIX  BEGINBMATRIX2  0  0 4  2  0 6  2  1 ENDBMATRIXBEGINBMATRIX2  4  6 0  2  2 0  0  1 ENDBMATRIX  LLT  BEGINBMATRIX1  0  0 2  1  0 3  1  1 ENDBMATRIXBEGINBMATRIX4  0  0 0  4  0 0  0  1 ENDBMATRIXBEGINBMATRIX1  2  3 0  1  1 0  0  1 ENDBMATRIX  UT D UENDALIGNED ENDEXAMPLEIF THE CHOLESKY FACTORIZATION DOES NOTEXIST SAY AS DETERMINED BY THE ALGORITHM BELOW THEN TO THEPRECISION AVAILABLE THE MATRIX B IS NOT POSITIVE DEFINITEBEGINEXAMPLE INDEXGAUSSIAN RANDOM NUMBER  IN A SIMULATION OF A SIGNAL PROCESSING ALGORITHM IT IS NECESSARY TO  GENERATE GAUSSIAN RANDOM VECTORS WITH COVARIANCE R  SYSTEM  LIBRARIES OFTEN PROVIDE GENERATORS WHICH SIMULATE INDEPENDENT  NC01 RANDOM VARIABLES  THESE CAN BE USED TO GENERATE  NC0R RANDOM VECTORS AS FOLLOWS  FIRST FACTOR R AS R  LLT WHERE L IS LOWER TRIANGULAR  FOR EACH RANDOM VECTOR DESIRED CREATEA VECTOR XBF OF NC01 INDEPENDENT RANDOM VARIABLES USING THEGAUSSIAN RANDOM NUMBER GENERATOR AND LET ZBF  L XBFTHEN SINCE EXBFXBFT  I EZBFZBFT  LEXBFXBFTLT  LLT  RSO ZBF HAS THE DESIRED COVARIANCEENDEXAMPLEBEGINEXAMPLE  THE CHOLESKY FACTORIZATION CAN BE USED TO SOLVE SYSTEMS OF  EQUATIONS  FOR THE EQUATION A XBF BBFWHERE A IS HERMITIAN AND POSITIVE DEFINITE WRITE A  LLH SOLUTION THEN REQUIRES SOLVING THE TWO SETS OF TRIANGULAR SYSTEMS BEGINALIGNEDLYBF  BBF LH XBF   YBFENDALIGNEDMUCH AS WAS DONE FOR THE LU DECOMPOSITIONENDEXAMPLEBEGINEXAMPLE APPLICATION OF CHOLESKY FACTORIZATION TO NORMAL  EQUATIONS  THE LEASTSQUARES SOLUTION REFEQLSMAT1 AH A XBF  AH BBFCAN BE SOLVED USING THE CHOLESKY FACTORIZATION WHERE AHA  LLHLET AHBBF  PBF  THEN FIRST SOLVE BY SUBSTITUTION LYBF  PBFTHEN SOLVE BY BACKSUBSTITUTION LH XBF  YBFSOLVING THE NORMAL EQUATIONS USING THE CHOLESKY FACTORIZATION ISSOMETIMES CALLED THE NORMAL EQUATION APPROACH INDEXLEASTSQUARESNORMAL  EQUATION APPROACHENDEXAMPLEWE WILL SEE IN SECTION REFSECQRFACT THAT THE QR DECOMPOSITION CANBE USED TO SOLVE LEASTSQUARES PROBLEMS  WHY THEN WOULD WE CONSIDERUSING THE CHOLESKY FACTORIZATION  IN FAVOR OF USING THE QR COMPUTINGAHA REQUIRED TO USE THE CHOLESKY FACTORIZATION REQUIRES A GOODDYNAMIC RANGE CAPABILITY ESSENTIALLY DOUBLE THE WORD SIZE FOR AFIXEDPOINT REPRESENTATION IN ORDER TO NOT BE HURT BY AN INCREASE INCONDITION NUMBER  ON THE OTHER HAND FOR AN MATSIZEMN MATRIXA IF M GG N THEN AHA AND ITS FACTORIZATIONS WILL REQUIRELESS STORAGE AND APPROXIMATELY HALF THE COMPUTATION OF THE QRREPRESENTATION  IN THIS CASE IF IT CAN BE DETERMINED THAT THE SYSTEMOF EQUATIONS IS SUFFICIENTLY WELL CONDITIONED SOLUTION USING CHOLESKYFACTORIZATION MAY BE JUSTIFIEDTHE CHOLESKY FACTORIZATION IS ALSO USED IN SQUARE ROOT KALMANINDEXSQUARE ROOT KALMAN FILTER FILTERING APPLICATIONS WHICH ARENUMERICALLY STABLE METHODS OF COMPUTING KALMAN FILTER UPDATES  SEEEG CITEVERHAEGEN SUBSECTIONALGORITHMS FOR COMPUTING THE CHOLESKY FACTORIZATIONTHERE ARE SEVERAL ALGORITHMS WHICH CAN BE USED TO COMPUTE THE CHOLESKYFACTORIZATION WHICH ARE MENTIONED FOR EXAMPLE IN CITEGVL  THEALGORITHM PRESENTED REQUIRES M33 FLOATING OPERATIONS AND REQUIRESNO ADDITIONAL STORAGE  THE ALGORITHM IS DEVELOPED RECURSIVELY  WRITE B  BEGINBMATRIX ALPHA  VBFH  VBF  B1 ENDBMATRIXAND NOTE THAT IT CAN BE FACTORED ASBEGINEQUATION B  BEGINBMATRIX SQRTALPHA  0  VBFSQRTALPHA   IN1 ENDBMATRIX BEGINBMATRIX1  0  0  B1  VBF VBFHALPHA ENDBMATRIXBEGINBMATRIX SQRTALPHA  VBFHSQRTALPHA  0  IN1ENDBMATRIXLABELEQBCHOLENDEQUATIONIF WE COULD FIND THE CHOLESKY FACTORIZATION OF B1  VBFVBFHALPHAAS G1G1H WE WOULD HAVE B  BEGINBMATRIX SQRTALPHA  0  VBFSQRTALPHA  G1ENDBMATRIX BEGINBMATRIX SQRTALPHA  VBFHSQRTALPHA   0  G1H ENDBMATRIX  GGHWE THEREFORE PROCEED RECURSIVELY DECOMPOSING B INTO SUCCESSIVELYSMALLER BLOCKS  THE SC MATLAB CODE IS DEMONSTRATED IN ALGORITHMREFALGCHOLESKY FOR DEMONSTRATION PURPOSES SINCE SC MATLAB HASA BUILTIN CHOLESKY FACTORIZATION VIA THE FUNCTION TT CHOLBEGINNEWPROGENVCHOLESKY    FACTORIZATIONCHOLESKYMCHOLESKYCHOLESKY    FACTORIZATIONENDNEWPROGENVBEGINEXERCISESITEM COMPUTE THE CHOLESKY FACTORIZATION OF  A  BEGINBMATRIX 464  62518  41822 ENDBMATRIXAS ALLT  THEN WRITE THIS AS A  UTDU WHERE U IS AN UPPER TRIANGULAR MATRIXWITH 1 ALONG THE DIAGONALITEM SHOW THAT REFEQBCHOL IS TRUEITEM GIVEN A ZEROMEAN DISCRETETIME INPUT SIGNAL FT WHICH WE  FORM INTO A VECTOR QBFT  BEGINBMATRIX FBART  FBART1  CDOTS   FBARTMENDBMATRIXTWE DESIRE TO FORM A SET OF OUTPUTS BBFT  BEGINBMATRIX B0T  B1T  CDOTS  BMTENDBMATRIXTBY BBFT  HQBFT THAT ARE UNCORRELATED THAT IS EBIT BBARJT  0 TEXT IF INEQ JLET R  EQBFT QBFBART BE THE CORRELATION MATRIX OF THE INPUTDATA  BEGINENUMERATEITEM DETERMINE THE MATRIX H WHICH DECORRELATES THE INPUT DATAITEM INTERPRET THE RESULTS AS A BANK OF  BACKWARD PREDICTORSENDENUMERATEITEM WRITE A SC MATLAB ROUTINE TT BACKCHOL WHICH FACTORS A  SYMMETRIC POSITIVE DEFINITE MATRIX AS A  UUH WHERE U IS AN  UPPER TRIANGULAR MATRIXITEM WRITE SC MATLAB ROUTINES TT FORSUBLYB AND TT    BACKSUBUYB TO SOLVE LYBF  BBF FOR A LOWER TRIANGULAR MATRIX    L AND U YBF  BBF FOR AN UPPER TRIANGULAR MATRIX U  ITEM DEVELOP A MEANS OF COMPUTING THE SOLUTION TO THE WEIGHTED    LEASTSQUARES PROBLEM REFEQWLS2 USING THE CHOLESKY    FACTORIZATION  ITEM LET X  XBF1 XBF2 LDOTS XBFN BE A SET OF    REALVALUED ZEROMEAN DATA WITH CORRELATION MATRIX RXX  FRAC1N XXTDETERMINE A TRANSFORMATION ON X THAT PRODUCES A DATA SET Y  SUCHTHAT RYY  FRAC1N YYTIS EQUAL TO AN IDENTITYENDEXERCISESSECTIONUNITARY MATRICES AND THE QR FACTORIZATIONLABELSECQRWE BEGIN WITH A DESCRIPTION OF THE Q IN THE QR FACTORIZATIONSUBSECTIONUNITARY MATRICESINDEXUNITARY MATRIX INDEXORTHOGONAL MATRIXBEGINDEFINITION  A MATSIZEMM MATRIX Q WITH COMPLEX ELEMENTS IS SAID TO BE  BF UNITARY IF  QHQ  IIF Q HAS REAL ELEMENTS AND QTQ  I THEN Q IS SAID TO BE AN BF  ORTHOGONAL MATRIXENDDEFINITIONFOR A UNITARY OR ORTHOGONAL MATRIX WE ALSO HAVE QQH  IBEGINLEMMA LABELLEMQRSAMENORMIF YBF  Q XBF FOR AN MATSIZEMM MATRIX Q  THEN YBF   XBF  FOR ALL XBF IN RBB IF AND ONLY IF  Q IS UNITARY WHERE THE NORM IS THE USUAL EUCLIDEAN NORMENDLEMMAA TRANSFORMATION WHICH DOES NOT CHANGE THE LENGTH OF A VECTOR IS SAIDTO BE BF ISOMETRIC OR LENGTHPRESERVING INDEXISOMETRIC THEPROOF OF THE LEMMA IS STRAIGHTFORWARD AND IS GIVEN AS AN EXERCISETHIS LEMMA ALLOWS US TO MAKE TRANSFORMATIONS ON VARIABLES EM  WITHOUT CHANGING THEIR LENGTH  THE LEMMA PROVIDES THE BASIS FORPARSEVALS THEOREM FOR FINITEDIMENSIONAL VECTORSBEGINLEMMA  LABELLEMQRSAMEFNORM  IF Y  QX FOR AN MATSIZEMM UNITARY  MATRIX Q THEN   YF  XFWHERE  CDOT F IS THE FROBENIUS NORMENDLEMMATHERE IS A USEFUL ANALOGY THAT CAN NOW BE INTRODUCEDBEGINDESCRIPTIONITEMHERMITIAN MATRICES SATISFYING AH  A ARE ANALOGOUS TO  REAL  NUMBERS NUMBERS WHOSE COMPLEX CONJUGATE IS EQUAL TO ITSELFITEMUNITARY MATRICES SATISFYING UHUI ARE ANALOGOUS TO COMPLEX  NUMBERS Z ON THE UNIT CIRCLE SATISFYING Z2  1ITEMORTHOGONAL MATRICES SATISFYING QTQ1 ARE ANALOGOUS TO THE  REAL NUMBERS Z  PM 1 SUCH THAT Z21ENDDESCRIPTIONTHE BILINEAR TRANSFORMATION INDEXBILINEAR TRANSFORMATIONBEGINEQUATION Z  FRAC1JR1JRLABELEQCAYLEYPREENDEQUATIONTAKES REAL NUMBERS R INTO THE UNIT CIRCLE Z1 MAPPING THENUMBER RINFTY TO Z1  ANALOGOUSLY BY EM CAYLEYS FORMULABEGINEQUATIONU  I  JRIJR1LABELEQCAYLEYFORMENDEQUATIONA HERMITIAN MATRIX R IS MAPPED TO A UNITARY MATRIX THAT DOES NOTHAVE AN EIGENVALUE OF 1INDEXCAYLEY TRANSFORMATIONSUBSECTIONTHE QR FACTORIZATIONLABELSECQRFACTIN THE QR FACTORIZATION AN MATSIZEMN MATRIX A IS WRITTEN AS A  QRWHERE Q IS AN MATSIZEMM UNITARY MATRIX AND R IS UPPERTRIANGULAR MATSIZEMN  AS DISCUSSED BELOW THERE ARE SEVERALWAYS IN WHICH THE QR FACTORIZATION CAN BE COMPUTED  IN THIS SECTIONWE FOCUS ON SOME OF THE USES OF THE FACTORIZATIONTHE MOST IMPORTANT APPLICATION OF QR IS TO FULLRANK LEASTSQUARESPROBLEMS  CONSIDER AXBF APPROX BBFWHERE M  N AND THE COLUMNS OF A ARE LINEARLY INDEPENDENT  INTHIS CASE THE PROBLEM IS SAID TO BE A FULLRANK LEASTSQUARESPROBLEM  THE SOLUTION XBFHAT WHICH MINIMIZES  A XBFHAT  BBF2 IS XBFHAT  AHA1AH BBFINDEXLEASTSQUARESQR SOLUTION HOWEVER THE CONDITION NUMBER OFAHA IS THE SQUARE OF THE CONDITION OF A SO DIRECT COMPUTATION ISNOT ADVISED  THIS POOR CONDITIONING CAN BE MITIGATED USING THE QRDECOMPOSITION  WHEN MN THE QR DECOMPOSITION CAN BE WRITTEN AS A  QR  QBEGINBMATRIXR1  ZEROBF ENDBMATRIXWHERE R1 IS MATSIZENN AND THE ZEROBF DENOTES AMATSIZEMNN BLOCK OF ZEROS  ALSO LETBEGINEQUATION QH BBF  BEGINBMATRIX CBF  DBF ENDBMATRIXLABELEQQBF1ENDEQUATIONWHERE CBF IS MATSIZEN1 AND DBF IS MATSIZEMN1  THENBEGINALIGNAXBF  BBF 22   QR XBF  BBF22 NONUMBER     QRXBF  QH BBF22LABELEQSAMENORMUSE  LEFT BEGINBMATRIX R1  0 ENDBMATRIXXBF   BEGINBMATRIXCBF  DBF ENDBMATRIXRIGHT22 LABELEQSN2   R1 XBF  CBF22   DBF22 NONUMBERENDALIGNWHERE REFEQSAMENORMUSE FOLLOWS SINCE BBF  QQH BBF ANDREFEQSN2 FOLLOWS FROM LEMMA REFLEMQRSAMENORM  SUCH PULLINGOF ORTHOGONAL MATRICES OUT OF THIN AIR TO SUIT SOME ANALYTICALPURPOSE IS QUITE COMMON  THE VALUE XBFHAT THAT MINIMIZESREFEQSN2 SATISFIES  R1 XBFHAT  CBFWHICH CAN BE READILY COMPUTED SINCE R1 IS A TRIANGULAR MATRIX  PUTANOTHER WAY SOLVING AHA XBF  AH BBF WITH A  QR LEADS TOBEGINEQUATION RH R XBF  RH QH BBF  FBFLABELEQQR1ENDEQUATIONWHERE FBF  RH QH BBF  EQUATION REFEQQR1 LEADS TO A PAIROF TRIANGULAR EQUATIONS WHICH CAN BE SOLVED AS WAS DONE FOR THE LUDECOMPOSITIOIN SOLVE RH YBF  FBFTHEN SOLVE RXBF  YBFIF A DOES NOT HAVE FULL COLUMN RANK OR IF M N COMPUTING THE QRDECOMPOSITION AND SOLVING LEASTSQUARES PROBLEMS THEREBY IS MOREDIFFICULT  THERE ARE ALGORITHMS TO COMPUTE THE QR DECOMPOSITION INTHIS CASE WHICH INVOLVE COLUMN PIVOTING  HOWEVER IN THISCIRCUMSTANCE IT IS RECOMMENDED TO USE THE SVD AND HENCE THESETECHNIQUES ARE NOT DISCUSSED HERE  SUBSECTIONQR FACTOR AND LEASTSQUARES FILTERSAS AN EXAMPLE OF THE USE OF THE QR FACTORIZATION CONSIDER THELEASTSQUARES PROBLEMBEGINEQUATION DBFK  AK HBFK  EBFKLABELEQQRLSFILT1ENDEQUATIONWHERE WE WISH TO MINIMIZE  EBFK 22 IN WHICH DBFK  BEGINBMATRIX D1  D2  CDOTS   DKENDBMATRIXT A  BEGINBMATRIX QBF1H  QBF2H  VDOTS  QBFKHENDBMATRIXSEE SECTION REFSECLSFILT  THE LEAST SQUARESSOLUTION CAN BE OBTAINED BY FINDING THE QR FACTORIZATION OF XK AK  QK BEGINBMATRIXR1K  ZEROBF ENDBMATRIXTHEN R1K HBF  LEFTQK DBFKRIGHTMATSIZEM1THUS WE CAN FIND HBF BY BACK SUBSTITUTIONSUBSECTIONCOMPUTING THE QR FACTORIZATIONLABELSECQRCOMPAT LEAST FOUR MAJOR WAYS OF COMPUTING THE QR FACTORIZATION ARE WIDELYREPORTED  THESE AREBEGINENUMERATEITEM THE GRAMSCHMIDT ALGORITHM  INDEXGRAMSCHMIDT PROCESSITEM THE MODIFIED GRAMSCHMIDT ALGORITHM  INDEXMODIFIED GRAMSCHMIDT THE GRAMSCHMIDT ALGORITHMS ARE DISCUSSED IN SECTION  REFSECGRAMSCHMITITEM HOUSEHOLDER TRANSFORMATIONSITEM GIVENS TRANSFORMATIONSENDENUMERATETHE GRAMSCHMIDT METHODS PROVIDE AN ORTHONORMAL BASIS SPANNING THECOLUMN SPACE OF A  THE QR FACTORIZATION USING THE HOUSEHOLDERTRANSFORMATION AND THE GIVENS ROTATIONS RELY ON SIMPLE INVERTIBLE ANDORTHOGONAL GEOMETRIC TRANSFORMATIONS  THE HOUSEHOLDER TRANSFORMATIONIS SIMPLY A REFLECTION OPERATION WHICH IS USED TO ZERO MOST OF ACOLUMN OF A MATRIX WHILE THE GIVENS ROTATION IS A SIMPLETWODIMENSIONAL ROTATION WHICH IS USED TO ZERO A PARTICULAR SINGLEELEMENT OF A MATRIX  THESE OPERATIONS MAY BE APPLIED IN SUCCESSION TOOBTAIN AN UPPER TRIANGULAR MATRIX IN THE QR FACTORIZATION  THEY MAYALSO BE USED IN OTHER CIRCUMSTANCES WHERE ZEROING PARTICULAR ELEMENTSOF A MATRIX WHILE PRESERVING THE EIGENVALUES OF THE MATRIX ISNECESSARY  OF THESE THE GRAMSCHMIDT METHODS ARE THE LEAST COMPLEXCOMPUTATIONALLY BUT ARE ALSO THE MOST POORLY CONDITIONEDSUBSECTIONHOUSEHOLDER TRANSFORMATIONSINDEXHOUSEHOLDER TRANSFORMATIONS RECALL FROM SECTIONREFSECPROJMAT THAT THE MATRIX WHICH PROJECTS ORTHOGONALLY ONTOLSPANVBF IS SEE REFEQPROJMAT1 PV  FRACVBF VBFHVBFHVBFAND THE ORTHOGONAL PROJECTION MATRIX IS PVPERP  IPVTHESE ARE SIMILAR TO THE HOUSEHOLDER TRANSFORMATION WITH RESPECT TO ANONZERO VECTOR VBF WHICH IS A TRANSFORMATION OF THE FORMBEGINEQUATIONBEGINSPLITHV  I  2 FRACVBF VBFHVBFH VBF  I  2PVENDSPLITLABELEQHVDEFENDEQUATIONIT IS STRAIGHTFORWARD TO SHOW THAT HV IS UNITARY AND HVHHVSEE EXERCISE REFEXHOUSE1  THE VECTOR VBF IS CALLED A BFHOUSEHOLDER VECTOR  OBSERVE HVVBF  VBF AND IF ZBF PERP VBFWITH RESPECT TO THE EUCLIDEAN INNER PRODUCT THAT HV ZBF  ZBFWRITE XBF AS XBF  PVBF XBF  PVBFPERP XBFTHEN HV XBF  PVBFPERP XBF  PVBF XBFWHICH CORRESPONDS TO A EM REFLECTION INDEXREFLECTION OF THEVECTOR XBF ACROSS THE SPACE PERPENDICULAR TO VBF AS SHOWN INFIGURE REFFIGHOUSEREFLECT  REFLECTING TWICE RETURNS THE ORIGINALPOINT HV2 XBF  XBF  AS AN OPERATOR WE CAN WRITE HV  PVBFPERP  PVBFBEGINFIGUREHTBPBEGINCENTERINPUTPICTUREDIRHOUSEHOLDER1ENDCENTERCAPTIONTHE HOUSEHOLDER TRANSFORMATION OF A VECTOR  LABELFIGHOUSEREFLECTENDFIGURETHE HOUSEHOLDER TRANSFORMATION CAN BE USED TO ZERO OUT ALL THEELEMENTS OF A VECTOR EXCEPT FOR ONE COMPONENT  THAT IS FOR A VECTORXBF  X1 X2 LDOTS XNT THERE IS A VECTOR VBF IN THEHOUSEHOLDER TRANSFORMATION HV SUCH THAT HV XBF  BEGINBMATRIXALPHA  0  VDOTS  0 ENDBMATRIXFOR SOME SCALAR ALPHA  SINCE HV IS UNITARY XBF2 HVXBF2 HENCE ALPHA  PM XBF2  ONE WAY OF VIEWINGTHE HOUSEHOLDER TRANSFORMATION IS AS A UNITARY TRANSFORMATION WHICHCOMPRESSES ALL OF THE ENERGY IN A VECTOR INTO A SINGLE COMPONENTZEROING OUT THE OTHER COMPONENTS OF THE VECTOR  TO FIND THE VECTORVBF IN THE TRANSFORMATION HV WRITE HV XBF  ALPHA BEGINBMATRIX 1  0  VDOTS  0 ENDBMATRIX ALPHA EBF1THENBEGINALIGNEDHVXBF  I  2FRACVBF VBFHVBFH VBF XBF  XBF  2FRACVBFH XBFVBF VBFHVBF  ALPHA EBF1ENDALIGNEDSO THAT LEFT2FRACVBFH XBFVBF VBFHRIGHTVBF  XBF  ALPHAEBF1THIS MEANS THAT VBF IS A SCALAR MULTIPLE OF XBF  ALPHAEBF1SINCE WE KNOW THAT ALPHA  PM  XBF2 AND SINCE SCALINGVBF BY A NONZERO SCALAR DOES NOT CHANGE THE HOUSEHOLDERTRANSFORMATION WE WILL TAKE VBF  XBF PM  XBF2 EBF1ALTHOUGH EITHER SIGN MAY BE TAKEN NUMERICAL CONSIDERATIONS SUGGEST APREFERRED VALUE  FOR REAL VECTORS IF XBF IS CLOSE TO A MULTIPLEOF EBF1 THEN VBF  XBF  SIGNX1  XBF2 EBF1 HAS ASMALL NORM WHICH COULD LEAD TO A LARGE RELATIVE ERROR IN THECOMPUTATION OF THE FACTOR 2VBFT VBF  THIS DIFFICULTY CAN BEAVOIDED BY CHOOSING THE SIGN BY VBF  XBF  SIGNX1  XBF2 EBF1BY THIS SELECTION  VBF  GEQ XBF  FOR COMPLEX VECTORSCHOOSING ACCORDING TO THE SIGN OF THE REAL PART IS APPROPRIATETHE OPERATION OF HV ON XBF CAN BE UNDERSTOOD GEOMETRICALLY USINGFIGURE REFFIGHOUSE1 WHERE THE SIGN HERE IS TAKEN SO THAT VBF XBF   XBF2 EBF1  SINCE VBF IS THE SUM OF TWOEQUALLENGTH VECTORS IT IS THE DIAGONAL OF AN EQUILATERALPARALLELOGRAM  THE OTHER DIAGONAL ORTHOGONAL TO THE FIRST  SEEEXERCISE REFEXPOTH RUNS FROMTHE VECTOR XBF TO TO THE VECTOR XBF2 EBF1  FROM THEFIGURE IT IS CLEAR THAT PVBF XBF  VBF2 AND PVBFPERPXBF  XBF  VBF2BEGINFIGUREHTBPBEGINCENTER  INPUTPICTUREDIRHOUSEHOLDERENDCENTERCAPTIONZEROING ELEMENTS OF A VECTOR BY A HOUSEHOLDER TRANSFORMATION  LABELFIGHOUSE1ENDFIGUREIN THE QR FACTORIZATION WE WANT TO CONVERT A TO AN UPPER TRIANGULARFORM USING A SEQUENCE OF ORTHOGONAL TRANSFORMATIONS  TO USE THEHOUSEHOLDER TRANSFORMATION TO COMPUTE THE QR FACTORIZATION OF AMATRIX FIRST CHOOSE A HOUSEHOLDER TRANSFORMATION H1 TO ZERO OUTALL BUT THE FIRST ELEMENT OF THE FIRST COLUMN OF A USING THE VECTORVBF1  FOR THE SAKE OF ILLUSTRATION LET A BE MATSIZE43THEN H1 A  BEGINBMATRIX ALPHA1  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES ENDBMATRIXWHERE TIMES INDICATES ELEMENTS OF THE MATRIX WHICH ARE NOT ZERO INGENERAL  LET Q1  H1  TO CONTINUE THE PROCESS FOR THE MATSIZE32 MATRIX ON THE LOWERRIGHT CHOOSE A HOUSEHOLDER TRANSFORMATION MATRIX H2 TO ZERO OUTTHE LAST 2 ELEMENTS USING THE VECTOR VBF2  COMBINING WITHTHE FIRST TRANSFORMATION IS DONE BY BEGINBMATRIX1  ZEROBF  0  H2 ENDBMATRIXBEGINBMATRIX ALPHA1  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES ENDBMATRIX Q2 Q1 A  BEGINBMATRIX ALPHA1  TIMES  TIMES 0  ALPHA2  TIMES 0  0  TIMES 0  0  TIMES ENDBMATRIXWHERE Q2  BEGINBMATRIX1  ZEROBF  0  H2 ENDBMATRIXFOR THE SAKE OF IMPLEMENTATION DESCRIBEDBELOW NOTE THAT Q2 CAN BE FORMED AS A HOUSEHOLDER MATRIX ASBEGINEQUATIONQ2  I  2FRACVBFTILDE2 VBFTILDE2HVBFTILDE2H VBFTILDE2 LABELEQHOUSE3ENDEQUATIONWHERE VBFTILDE2  BEGINBMATRIX 0  VBF2 ENDBMATRIXTHE LAST TWO ELEMENTS IN THE THIRD COLUMN CAN BE REDUCED WITH A THIRDHOUSEHOLDER TRANSFORMATION H3  IN CONJUNCTION WITH THE OTHERELEMENTS OF THE MATRIX THIS CAN BE WRITTEN AS BEGINBMATRIX1  0  ZEROBF  0  1  0  0  0  H3 ENDBMATRIXBEGINBMATRIX ALPHA1  TIMES  TIMES 0  ALPHA2  TIMES 0  0  TIMES 0  0  TIMES ENDBMATRIX Q3 Q2 Q1 A  BEGINBMATRIX ALPHA1  TIMES  TIMES 0  ALPHA2  TIMES 0  0  ALPHA3 0  0  TIMES ENDBMATRIXWHERE Q3  BEGINBMATRIX1  0  ZEROBF  0  1  0  0  0  H3ENDBMATRIX  I  2 FRACVBFTILDE3 VBFTILDE3HVBFTILDE3H  VBFTILDE3 AND VBFTILDE3  BEGINBMATRIX 0  0  VBF3 ENDBMATRIXSINCE H2 AND H3 ARE ORTHOGONAL SO ARE Q2 AND Q3 SEEEXERCISE REFEXSTACKORTHOG AND SO IS QH  Q3 Q2 Q1  THUS AHAS BEEN REDUCED TO THE PRODUCT OF AN ORTHOGONAL MATRIX TIMES ANUPPER TRIANGULAR MATRIX A  QR  Q1 Q2 Q3 RFOR A GENERAL MATSIZEMN MATRIX COMPUTATION OF THE QR ALGORITHMINVOLVES FORMING N ORTHOGONAL MATRICES QJ J12LDOTSNTHEN Q  Q1 Q2 CDOTS QNWHERE  QJ  I  2 FRACVBFTILDEJ VBFTILDEJHVBFTILDEJH  VBFTILDEJAND VBFTILDEJ  UNDERBRACE00LDOTS0J1VBFJTTSUBSECTIONALGORITHMS FOR HOUSEHOLDER TRANSFORMATIONSIN THIS SECTION SOME SAMPLE SC MATLAB CODE IS DEVELOPED TO COMPUTETHE QR DECOMPOSITION USING HOUSEHOLDER TRANSFORMATIONS  THE CODE ISFOR DEMONSTRATION PURPOSES ONLY SINCE SC MATLAB HAS THE FUNCTIONTT QR BUILTININ THE INTEREST OF EFFICIENCY THE HOUSEHOLDER TRANSFORMATION MATRIXQ IS NOT EXPLICITLY FORMED  RATHER THAN EXPLICITLY FORMING HVAND THEN MULTIPLYING HV A WE NOTE THATBEGINEQUATION HV A  LEFTI  2 FRACVBF VBFHVBFH VBFRIGHT A  A BETA VBF WBFHLABELEQHOUSELEFTENDEQUATIONWHERE BETA  2VBFH VBF AND WBF  AH VBF  IT IS OFTEN THECASE THAT ONLY THE R MATRIX IS EXPLICITLY NEEDED SO THE Q ISREPRESENTED IMPLICITLY BY THE SEQUENCE OF VBFJ VECTORS FROM WHICHQ CAN BE COMPUTED AS DESIRED  ALGORITHM REFALGHOUSELEFTILLUSTRATES A FUNCTION WHICH APPLIES A HOUSEHOLDER TRANSFORMATIONHV REPRESENTED ONLY BY THE HOUSEHOLDER VECTOR VBF ON THE LEFTOF A AS HV A AND ALSO SHOWS THE FUNCTION TT MAKEHOUSE WHICHCOMPUTES THE HOUSEHOLDER VECTOR VBF  ALSO SHOWN IS THE FUNCTIONTT HOUSERIGHT WHICH APPLIES THE HOUSEHOLDER TRANSFORMATION ON THERIGHT TO ZERO OUT ROWS OF ABEGINNEWPROGENVHOUSEHOLDER TRANSFORMATION FUNCTIONS 1 COMPUTE    VBF 2 HV A  GIVEN  VBF AND 3 COMPUTE A HV GIVEN VBF  HOUSELEFTCOMPUTE PROTECTHPROTECTVPROTECT GIVEN VBFMAKEHOUSEMHOUSELEFTMHOUSERIGHTMENDNEWPROGENVBEGINEXAMPLE  LET  A  BEGINBMATRIX1  2  3 4  5  6 6  7  8 ENDBMATRIXTHEN THE SC MATLAB FUNCTION CALLS TT VL  MAKEHOUSEA1 ANDTT VR  MAKEHOUSEA1 RETURN THE VECTORS VL  BEGINBMATRIX828011  4  6 ENDBMATRIXT QQUADVR  BEGINBMATRIX474166  2  3 ENDBMATRIXTTHEN HVA CAN BE COMPUTED USING TT HOUSELEFTAVL AND A HVCAN BE COMPUTED FROM TT HOUSERIGHTAVR  THE RESULTS AREBEGINALIGNED TT HOUSELEFTAVL  BEGINBMATRIX728011  879108  10302 0  0213011  0426022 0  0819517  163903 ENDBMATRIX TT HOUSERIGHTAVR  BEGINBMATRIX374166  0  0 855236  0294503  194175 117595  0490838  323626 ENDBMATRIXENDALIGNEDENDEXAMPLEALGORITHM REFALGHOUSE1 COMPUTES THE QR FACTORIZATION USING THESIMPLIFICATIONS NOTED HERE  THE RETURN VALUES ARE THE MATRIX R ANDTHE VECTOR OF VBF VECTORS  THE COMPLEXITY OF THE ALGORITHM ISAPPROXIMATELY 2N2MN3 FLOATING OPERATIONSBEGINNEWPROGENVQR FACTORIZATION VIA HOUSEHOLDER TRANSFORMATIONSQRHOUSEMHOUSE1QR FACTORIZATION VIA HOUSEHOLDER  TRANSFORMATIONSENDNEWPROGENVIN ORDER TO SOLVE THE LEASTSQUARES EQUATION AS DESCRIBED ABOVE WE MUST BEABLE TO COMPUTE QH BBF  SINCE Q  Q1 Q2 CDOTS QN AND EACHQ IS HERMITIAN SYMMETRIC QH BBF  QNH QN1H CDOTS Q1H BBFWHICH MAY BE ACCOMPLISHED CONCEPTUALLY USING THE FOLLOWINGALGORITHM WHICH OVERWRITES BBF WITH QH BBFBEGINPROGTABSFOR J1N    BBF  QJ BBF ENDENDPROGTABSTHE MULTIPLICATION CAN BE ACCOMPLISHED WITHOUT EXPLICITLY FORMING THEQJ MATRICES USING THE IDEA SHOWN IN REFEQHOUSELEFTCOMPUTATION OF QH BBF IS THUS ACCOMPLISHED AS SHOWN IN ALGORITHMREFALGQRQTB BEGINNEWPROGENVCOMPUTATION OF QH BBFQRQTBMQRQTBCOMPUTATION OF QH BBFENDNEWPROGENVBEGINEXAMPLE  SUPPOSE IT IS DESIRED TO FIND THE LEASTSQUARES SOLUTION TO BEGINBMATRIX7 8  8 8 6  2 1 7  3 0 7  3 6 9  5ENDBMATRIXXBF  BEGINBMATRIX47  26 24 23  39  ENDBMATRIXUSING TT VR  QRHOUSEA WE OBTAIN V  BEGINBMATRIXHFILL 192474 HFILL 0  HFILL 0 HFILL 8  HFILL 127989  HFILL  0 HFILL 1  HFILL 58844  HFILL 61096 HFILL 0  HFILL 7    HFILL 23046 HFILL 6  HFILL 23065  HFILL 19142 ENDBMATRIXQQUADQQUADR  BEGINBMATRIXHFILL 122474  HFILL 134722   HFILL 85732 HFILL 0   HFILL  98742    HFILL 48105HFILL 0  HFILL 0  HFILL    37893 HFILL  0HFILL 0HFILL  0 HFILL 0HFILL 0 HFILL 0ENDBMATRIXUSING TT QRQTBBV WE OBTAINQH BBF  BEGINBMATRIXHFILL 649115  HFILL 341800  HFILL  113680  0  0 ENDBMATRIX THE LEASTSQUARES SOLUTION COMES FROM SOLVING THE MATSIZE33UPPERTRIANGULAR SYSTEM OF EQUATIONS USING BACKSUBSTITUTION BEGINBMATRIX HFILL 122474  HFILL 134722 HFILL   85732 HFILL 0HFILL 98742 HFILL    48105 HFILL 0HFILL 0   37893ENDBMATRIXXBFHAT  BEGINBMATRIXHFILL 649115  HFILL 341800  HFILL  113680ENDBMATRIXWHICH LEADS TO XBFHAT  BEGINBMATRIX1  2  3 ENDBMATRIXENDEXAMPLEWHERE THE Q MATRIX IS EXPLICITLY DESIRED FROM V IT CAN BE COMPUTED BYBACKWARD ACCUMULATION  TO COMPUTE Q  Q1Q2LDOTS QR WE ITERATEAS FOLLOWS BEGINALIGNEDQ0  I Q1  QR Q0  Q2  QR1 Q1 VDOTS Q  QR  Q1 QR1ENDALIGNEDAN ALGORITHM IMPLEMENTING THIS IS SHOWN IN ALGORITHM REFALGMAKEHOUSEQBEGINNEWPROGENVCOMPUTATION OF Q FROM VQRMAKEQMMAKEHOUSEQCOMPUTATION OF Q FROM VENDNEWPROGENVSUBSECTIONQR FACTORIZATION USING GIVENS ROTATIONSLABELSECGIVENSINDEXGIVENS ROTATIONSUNLIKE THE HOUSEHOLDER TRANSFORMATION WHICH ZEROS OUT ENTIRE COLUMNSAT A STROKE THE GIVENS ROTATION MORE SELECTIVELY ZEROS ONE ELEMENT ATA TIME USING A ROTATIONA TWODIMENSIONAL ROTATION BY AN ANGLE THETA IS ILLUSTRATED INFIGURE REFFIGGIVROT1A  THE FIGURE DEMONSTRATES THAT THE POINT10 IS ROTATED INTO THE POINT COS THETASINTHETA AND THE POINT 01 IS ROTATED INTO THE POINT SIN THETA COSTHETA  BY THESE POINTS WE IDENTITY THAT A MATRIX GTHETA WHICHROTATES XYT ISBEGINEQUATION GTHETA  BEGINBMATRIX  COS THETA  SIN THETA  SIN THETA   COS THETA  ENDBMATRIX LABELEQGIVENS1ENDEQUATIONTHE ROTATION MATRIX IS ORTHOGONAL GTHETATGTHETA  I  ITSHOULD BE CLEAR THAT ANY POINT XY IN TWO DIMENSIONS CAN BEROTATED BY SOME ROTATION MATRIX G SO THAT ITS SECOND COORDINATE ISZERO  THIS IS ILLUSTRATED IN FIGURE REFFIGGIVROT1B  FOR AVECTOR XBF  X YT ITS SECOND COORDINATE CAN BE ZEROED BYMULTIPLICATION BY THE ORTHOGONAL MATRIX GTHETA WHEREINDEXROTATION MATRIXBEGINEQUATIONTHETAXY  TAN1 FRACYXLABELEQGIVETHETAENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODESUBFIGUREA GENERAL ROTATIONINPUTPICTUREDIRROT1QQUADQQUADSUBFIGUREROTATE THE SECOND COORDINATE TO ZEROINPUTPICTUREDIRROT2    CAPTIONTWODIMENSIONAL ROTATION    LABELFIGGIVROT1  ENDCENTERENDFIGUREIN THE GIVENS ROTATION APPROACH TO THE QR FACTORIZATION A MATRIX AIS ZEROED OUT ONE ELEMENT AT A TIME STARTING AT THE BOTTOM OF THEFIRST COLUMN AND WORKING UP THE COLUMNS  TO ZERO AIK WE USE X AJK AND YAIK APPLYING THE MATSIZE22 ROTATION MATRIXACROSS THE JTH AND ITH ROWS OF A  SUCH A ROTATION MATRIX ISCALLED A EM GIVENS ROTATION  WE WILL DENOTE BY GTHETAIKJTHE ROTATION MATRIX WHICH ZEROS AIK  FOR BREVITY WE WILL ALSOWRITE GIKJ  IN THE QR FACTORIZATION A SEQUENCE OF THESEROTATION MATRICES ARE USED  A SEQUENCE OF MATRICES PRODUCED BYSUCCESSIVE OPERATION OF GIVENS ROTATIONS MIGHT HAVE THE FOLLOWINGFORM WHERE THE CONVENTION OF TAKING JI1 IS USED  THE ROTATION ISSHOWN ABOVE THE ARROW AND THE ROWS AFFECTED BY THE PRECEDINGTRANSFORMATION ARE SHOWN IN BOLDBEGINEQUATIONBEGINALIGNED BEGINBMATRIX TIMESTIMESTIMES  TIMES  TIMES  TIMES TIMES  TIMES  TIMES  TIMES  TIMES  TIMES ENDBMATRIXSTACKRELG413LONGRIGHTARROWBEGINBMATRIX TIMESTIMESTIMES  TIMES  TIMES  TIMES TIMESBF  TIMESBF  TIMESBF  ZEROBF  TIMESBF  TIMESBF ENDBMATRIXSTACKRELG312LONGRIGHTARROWBEGINBMATRIX TIMESTIMESTIMES  TIMESBF  TIMESBF  TIMESBF ZEROBF  TIMESBF  TIMESBF  0  TIMES  TIMES ENDBMATRIX STACKRELG211LONGRIGHTARROWBEGINBMATRIX TIMESBF TIMESBF TIMESBF  ZEROBF  TIMESBF  TIMESBF 0  TIMES  TIMES  0  TIMES  TIMES ENDBMATRIX STACKRELG423LONGRIGHTARROWBEGINBMATRIX TIMES TIMESTIMES  0  TIMES  TIMES 0  TIMESBF  TIMESBF  0  ZEROBF  TIMESBF ENDBMATRIX STACKRELG322LONGRIGHTARROWBEGINBMATRIX TIMES TIMESTIMES  0  TIMESBF  TIMESBF 0  ZEROBF  TIMESBF  0  0  TIMES ENDBMATRIX STACKRELG433LONGRIGHTARROWBEGINBMATRIX TIMES TIMESTIMES 0  TIMES  TIMES 0  0  TIMESBF 0  0  ZEROBF ENDBMATRIXENDALIGNEDLABELEQGIV2ENDEQUATIONTHE TWODIMENSIONAL ROTATION REFEQGIVENS1 CAN BE MODIFIED TOFORM GTHETAIKJ BY DEFININGBEGINEQUATIONBEGINALIGNEDGTHETAIKJ  BEGINBMATRIX1 CDOTS  0  CDOTS  0  CDOTS  0 VDOTS  DDOTS  VDOTS   VDOTS  VDOTS 0  CDOTS   C CDOTS  S CDOTS  0 VDOTS  VDOTS DDOTS  VDOTS  VDOTS 0  CDOTS  S  CDOTS  C  CDOTS 0 VDOTS  VDOTS  VDOTS  DDOTS VDOTS 0  CDOTS  SMASHLOWER15EMVBOXHBOX0HBOXRULE0MM15EMJ  CDOTS  SMASHLOWER15EMVBOXHBOX0HBOXRULE0MM15EMI CDOTS 1ENDBMATRIX BEGINMATRIX   J   I   ENDMATRIX  BEGINMATRIX  ENDMATRIXENDALIGNEDLABELEQGIVENGENDEQUATIONWHERE C  COS THETA AND S  SIN THETA  AS IS APPARENT FROMTHE FORM OF GTHETAIKJ THE OPERATION GTHETAIKJA SETSTHE IKTH ELEMENT TO ZERO AND MODIFIES ITH AND JTH ROWS OFA LEAVING THE OTHER ROWS OF A UNMODIFIED  THE VALUE OF THETAIN GTHETAIKK IS DETERMINED FROM XY  AJKAIK INREFEQGIVETHETA  AS IS APPARENT BY STUDYING REFEQGIV2TAKING JI1 MAKES IT SO THAT THE DIAGONALIZATION ALREADYACCOMPLISHED IN PRIOR COLUMNS IS NOT AFFECTED BY GIVENS ROTATIONS ONLATER COLUMN  SINCE THIS IS THE MOST COMMON CASE WE WILL HENCEFORTHUSE THE ABBREVIATED NOTATION GIK OR GTHETAIK FORGTHETAIKJFOR THE MATSIZE43 EXAMPLE THE FACTORIZATION IS ACCOMPLISHED BYG41G31G21G42G32G43A RTHE Q MATRIX IS THUS OBTAINED ASBEGINALIGNEDQ  G41G31G21G42G32G43T  G43TG32TG42T G21T G31T G41TENDALIGNEDBEGINEXAMPLE  LET  A  BEGINBMATRIX1 2 3 4 1 3ENDBMATRIXA ROTATION MATRIX TO ZERO THE 31 ELEMENT THAT MODIFIES THE LASTTWO ROWS OF A IS G31  G312  BEGINBMATRIX1 0 0  0 09487 03162  0   03162  09487 ENDBMATRIXTHEN  GA  BEGINBMATRIX1 2 31623  47434 0     15811 ENDBMATRIXENDEXAMPLESUBSECTIONALGORITHMS FOR QR FACTORIZATION USING GIVENS ROTATIONSSEVERAL ASPECTS OF THE MATHEMATICS OUTLINED ABOVE FOR GIVENS ROTATIONSMAY BE STREAMLINED FOR A NUMERICAL IMPLEMENTATION  EXPLICITCOMPUTATION OF THETA IS NOT NECESSARY WHAT ARE NEEDED IS COSTHETA AND SIN THETA WHICH MAY BE DETERMINED FROM XYWITHOUT ANY TRIGONOMETRIC FUNCTIONS COS THETA  COS TAN LEFTFRACYXRIGHT FRACXSQRTX2  Y2 QQUAD SIN THETA FRACYSQRTX2Y2SEE ALGORITHM REFALGQRTHETA FOR A NUMERICALLY STABLE METHOD OFCOMPUTING THESE QUANTITIES BEGINNEWPROGENVFIND C AND S FOR A GIVENS ROTATION    QRTHETAMQRTHETAFIND C AND S FOR A GIVENS ROTATIONENDNEWPROGENVIN COMPUTING THE MULTIPLICATION GIK A IT IS CLEARLY MUCH MOREEFFICIENT TO ONLY MODIFY ROWS I AND K OF THE PRODUCT  AN EXPLICITQ MATRIX IS NEVER CONSTRUCTED  INSTEAD THE COS THETA AND SINTHETA INFORMATION IS SAVED  IT WOULD ALSO BE POSSIBLE TO REPRESENTBOTH NUMBERS USING A SINGLE QUANTITY AND STORE THE Q MATRIXINFORMATION IN THE LOWER TRIANGLE  HOWEVER IN THE INTEREST OF SPEEDTHIS IS NOT DONE  AN ALGORITHM TO COMPUTE THE QR FACTORIZATION ISSHOWN IN ALGORITHM REFALGQRGIVENS AND ALGORITHM REFALGQRQTBGIVCOMPUTES QH BBF FOR USE IN SOLVING LEASTSQUARES PROBLEMSFINALLY FOR THOSE INSTANCES IN WHICH IT IS NEEDED ALGORITHMREFALGMAKEGIVQ COMPUTES Q FROM THE THETA INFORMATION BYCOMPUTING Q  GM1TGM11TCDOTS G21TGM2TGM12TCDOTSGMNTWITH THE MULTIPLICATION DONE FROM LEFT TO RIGHTBEGINNEWPROGENVQR FACTORIZATION USING GIVENS ROTATIONS    QRGIVENSMQRGIVENSQR FACTORIZATION USING GIVENS      ROTATIONSENDNEWPROGENVBEGINNEWPROGENVCOMPUTATION OF QH BBF FOR THE GIVENS ROTATION    FACTORIZATIONQRQTBGIVMQRQTBGIVCOMPUTATION OF QH BBF FOR THE  GIVENS ROTATION FACTORIZATIONENDNEWPROGENVBEGINNEWPROGENVCOMPUTATION OF Q FROM THE THETAQRMAKEQGIVMMAKEGIVQCOMPUTATION OF Q FROM THETAENDNEWPROGENVSUBSECTIONSOLVING LEASTSQUARES PROBLEMS USING GIVENS ROTATIONSGIVENS ROTATIONS CAN BE USED TO SOLVE LEASTSQUARES PROBLEMS IN A WAYTHAT IS WELLSUITED FOR PIPELINED IMPLEMENTATION IN VLSICITEPROAKISRADER  INDEXLEASTSQUARESVLSIAPPROPRIATE  ALGORITHMS REWRITE THE EQUATION A XBF APPROX BBFAS A  BBFBEGINBMATRIX XBF  1 ENDBMATRIX APPROX 0LET THIS BE WRITTEN AS B HBF APPROX 0 WHERE B  ABBF AND HBFT  XBFT 1  THEN THELEASTSQUARES SOLUTION IS THE ONE WHICH MINIMIZES  BHBF22 HBFH BH B HBF  SINCE MULTIPLICATION BY AN ORTHOGONAL MATRIX DOESNOT CHANGE THE NORM  QBHBF 22   BHBF22 FOR ANORTHOGONAL MATRIX Q  THE MATRIX Q CAN BE SELECTED AS A GIVENSROTATION WHICH SELECTIVELY ZEROS OUT ELEMENTS OF THE MATRIX B  BYTHIS MEANS WE CAN TRANSFORM THE PROBLEM SUCCESSIVELY AS BEGINALIGNEDBHBF APPROX 0 Q1BHBF APPROX 0 Q2Q1BHBF APPROX 0 Q2Q1BHBF APPROX 0 LDOTSQP CDOTS Q2Q1B HBF APPROX 0ENDALIGNEDWITH AN APPROPRIATELYCHOSEN SEQUENCE OF QI MATRICES THE RESULT ISTHAT QP CDOTS Q2Q1B IS MOSTLY UPPER TRIANGULAR SO THAT WE OBTAINA SET OF EQUATION OF THE FOLLOWING FORM BEGINBMATRIX AHAT11  AHAT12 AHAT13  CDOTS   AHAT1N  BHAT1  AHAT22  AHAT23 CDOTS  AHAT2N  BHAT2  AHAT33 CDOTS  AHAT3N  BHAT3 VDOTS  AHATNN  BHATN TIMES  TIMES  TIMES  TIMES TIMES  BHATN1 TIMES  TIMES  TIMES  TIMES TIMES  BHATN2 VDOTS  ENDBMATRIXBEGINBMATRIX X1  X2  X3  VDOTS  XN  1 ENDBMATRIXAPPROX BEGINBMATRIX0  0  0  VDOTS  0  VDOTSENDBMATRIXPRACTICALLY SPEAKING MULTIPLICATION BY THE ORTHOGONAL MATRICES CANSTOP WHEN THE TOP N ROWS ARE MOSTLY TRIANGULARIZED AS SHOWN  WHILEIT WOULD BE POSSIBLE TO COMPLETE THE QR FACTORIZATION TO ZERO THELOWER PORTION OF THE MATRIX THE PART INDICATED WITH TIMESS THISIS NOT NECESSARY SINCE THE STRUCTURE ALLOWS THE SOLUTION TO BEOBTAINED  FROM THIS THE LEASTSQUARE SOLUTION ISBEGINALIGNEDXN  FRACBHATNAHATNN XN1  FRACBHATN1  AHATN1N XNAHATN1N1  VDOTS XI  FRACBHATI  SUMJI1N AIJXJAHATII VDOTS X1  FRACBHAT1  SUMJ2N A1JXJAHAT11ENDALIGNEDSUBSECTIONGIVENS ROTATIONS VIA CORDIC ROTATIONSINDEXCORDIC ROTATIONSFOR HIGHSPEED REALTIME APPLICATIONS IT MAY BE NECESSARY TO GO WITHPIPELINED AND PARALLEL ALGORITHMS FOR QR DECOMPOSITION  THE METHODKNOWN AS CORDIC ROTATIONS PROVIDES FOR PIPELINED IMPLEMENTATIONS OFTHE GIVENS ROTATIONS WITHOUT THE NEED TO COMPUTE TRIGONOMETRICFUNCTIONS OR SQUAREROOTS  CORDIC IS AN ACRONYM FOR COORDINATEROTATION DIGITAL COMPUTATION  CORDIC METHODS HAVE ALSO BEEN APPLIEDTO A VARIETY OF OTHER SIGNAL PROCESSING PROBLEMS INCLUDING DFTSFFTS DIGITAL FILTERING AND ARRAY PROCESSING  A SURVEY ARTICLE WITHA VARIETY OF REFERENCES IS CITEHU1992  A DETAILED APPLICATION OFCORDIC TECHNIQUES TO ARRAY PROCESSING USING A VLSI HARDWAREIMPLEMENTATION INCLUDING SOME VERY CLEVER DESIGNS FOR SOLUTION OFLINEAR EQUATIONS APPEARS IN CITERADER1996THE FUNDAMENTAL STEP IN GIVENS ROTATIONS IS THE TWO DIMENSIONALROTATIONBEGINEQUATION BEGINBMATRIX X  Y ENDBMATRIX  BEGINBMATRIX COS  THETA  SIN THETA  SIN THETA  COS THETAENDBMATRIXBEGINBMATRIXX  Y ENDBMATRIXLABELEQCORDIC1ENDEQUATIONWHERE THETA IS CHOSEN SO THAT Y0  THIS TRANSFORMATION ISAPPLIED SUCCESSIVELY TO APPROPRIATE PAIRS OF ROWS TO OBTAIN THE QRFACTORIZATION  SINCE IT IS REPEATEDLY USED IT IS IMPORTANT TO MAKETHE COMPUTATION AS EFFICIENT AS POSSIBLE  THE ROTATION INREFEQCORDIC1 CAN BE REWRITTEN ASBEGINEQUATION BEGINBMATRIXX  Y ENDBMATRIX  COS THETABEGINBMATRIX  1  TAN THETA  TAN THETA  1 ENDBMATRIXBEGINBMATRIX X  Y ENDBMATRIXLABELEQCORDIC0ENDEQUATIONWHICH STILL REQUIRES FOUR MULTIPLICATIONS  HOWEVER IF THE ANGLETHETA IS SUCH THAT TAN THETA IS A POWER OF TWO THEN THEMULTIPLICATION CAN BE ACCOMPLISHED USING ONLY BITSHIFT OPERATIONSA GENERAL ANGLE CAN BE CONSTRUCTED AS A SERIES OF ANGLES WHOSETANGENTS ARE POWERS OF TWO THETA  SUMI0INFTY RHOI THETAIWHERE RHOI  PM 1 AND THETAI IS CONSTRAINED SO THAT TANTHETAI  2I  IN PRACTICE THE SUM IS TRUNCATED AFTER A FEWTERMS USUALLY ABOUT FIVE OR SIX THETA APPROX SUMI0ITEXT MAX RHOI THETAITHE POWEROF TWO ANGLES FOR CORDIC ROTATIONS ARE SHOWN TABLEREFTABCORDIC UP TO THETA6  HIGHER ACCURACY IN THEREPRESENTATION CAN BE OBTAINED BY TAKING MORE TERMS ALTHOUGH FOR MOSTPRACTICAL PURPOSES UP TO FIVE TERMS IS OFTEN ADEQUATE  BEGINTABLEHTBP    BEGINCENTER      LEAVEVMODE      BEGINTABULARLCRR HLINEI  TAN THETAI  THETAI DEGREES MULTICOLUMN1CKAPPAI  HLINE 0  1  45                    070711 EXMATSP1  FRAC12   265605  063245EXMATSP2  FRAC14   140362  061357 EXMATSP3  FRAC18   712502  060883 EXMATSP4  FRAC116  357633  060764 EXMATSP5  FRAC132  178991  060728 EXMATSP6  FRAC164  089517  060726 EXMATSP HLINE      ENDTABULAR      CAPTIONPOWEROFTWO ANGLES FOR CORDIC COMPUTATIONS      LABELTABCORDIC    ENDCENTER  ENDTABLEBEGINEXAMPLE  AN ANGLE SUCH AS 37CIRC CAN BE REPRESENTED USING THE ANGLES IN  TABLE REFTABCORDIC AS 37 APPROX THETA0  THETA1  THETA2  THETA3  THETA4 THETA5  THETA6  3691832AN EFFICIENT REPRESENTATION IS SIMPLY THE SEQUENCE OF SIGNS 37SIM 1111111ENDEXAMPLETHE ROTATION BY THETA IN REFEQCORDIC1IS ACCOMPLISHEDSTAGEWISE BY A SERIES OF EM MICROROTATIONS  WHAT MAKES THIS MOREEFFICIENT IS THE FACT THAT THE FACTORS COS THETAI FROM EACHMICROROTATION CAN BE COMBINED TOGETHER INTO A PRECOMPUTED CONSTANT KAPPAITEXT MAX  PRODI0ITEXT MAX COS THETAITABLE REFTABCORDIC SHOWS THE VALUES OF KAPPA FOR THE FIRST FEWVALUES OF ITEXT MAX  THE MICROROTATIONS RESULT IN A SERIES OFINTERMEDIATE RESULTS  IN A CORDIC IMPLEMENTED IN IMAX STAGESTHE FOLLOWING RESULTS ARE OBTAINED BY SUCCESSIVE APPLICATION OFREFEQCORDIC0 BEGINALIGNEDBEGINBMATRIX X0  Y0 ENDBMATRIX  KAPPABEGINBMATRIXX  Y ENDBMATRIX BEGINBMATRIX X1  Y1 ENDBMATRIX  BEGINBMATRIX  X0  Y0 ENDBMATRIX  RHO0 20 BEGINBMATRIX  Y0  X0 ENDBMATRIX BEGINBMATRIX X2  Y2 ENDBMATRIX  BEGINBMATRIX  X1  Y1 ENDBMATRIX  RHO1 21 BEGINBMATRIX  Y1  X1 ENDBMATRIX BEGINBMATRIX X3  Y3 ENDBMATRIX  BEGINBMATRIX  X2  Y2 ENDBMATRIX  RHO2 22 BEGINBMATRIX  Y2  X2 ENDBMATRIX VDOTS BEGINBMATRIX X  Y ENDBMATRIX  BEGINBMATRIX  XIMAX  YIMAX ENDBMATRIX  RHOIMAX2IMAX BEGINBMATRIX   YIMAX  XIMAX ENDBMATRIX ENDALIGNEDTHE EFFECT OF MULTIPLICATION BY KAPPA IS TO NORMALIZE THE VECTOR SOTHAT THE FINAL VECTOR XYT HAS THE SAME LENGTH AS XYTIN CIRCUMSTANCES WHERE THE ANGLE OF THE VECTOR IS IMPORTANT BUT NOTITS LENGTH THE FIRST STEP MAY BE ELIMINATEDWHEN DOING ROTATION FOR THE QR ALGORITHM THE ANGLE THETA THROUGHWHICH TO ROTATE IS DETERMINED BY THE FIRST ELEMENT ON EACH OF THE TWOROWS BEING ROTATED  THESE ELEMENTS ARE REFERRED TO AS THE EM  LEADER OF THE PAIR OF ROWS  THE REST OF THE ELEMENTS ON THE ROWARE ROTATED AT AN ANGLE DETERMINED BY THE LEADER  FOR THE REGULARGIVENS ROTATION IT IS NECESSARY TO COMPUTE THE ANGLE WHICH AT AMINIMUM REQUIRES COMPUTATION OF A SQUARE ROOT  HOWEVER FOR THECORDIC IMPLEMENTATION IT IS POSSIBLE TO DETERMINE THE ANGLE TO ROTATETHROUGH IMPLICITLY USING THE MICROROTATIONS SIMPLY BY EXAMINING THESIGNS OF THE COMPONENTS OF THE LEADER  THE GOAL IS TO ROTATE A VECTORXBFT  XY TO A VECTOR X0  IF XBF IS IN QUADRANT I ORQUADRANT III THEN THE ROTATION IS NEGATIVE  IF XBF IS IN QUADRANTII OR QUADRANT IV THEN THE ROTATION IS POSITIVE  THE SIGN OF THEMICROROTATION IS DETERMINED BYBEGINEQUATION RHOI  SIGNXI1SIGNYI1LABELEQCORDIC2ENDEQUATIONIN A PIPELINES IMPLEMENTATION OF THE CORDIC ARCHITECTURE A SEQUENCEOF 2VECTORS FROM A PAIR OF ROWS OF THE MATRIX A ARE PASSED THROUGHTHE STRUCTURE SHOWN  AS THE FIRST VECTOR FROM EACH ROW  THE LEADER IS PASSED THE MICROROTATION ANGLE IS COMPUTED ACCORDING TOREFEQCORDIC2  THIS INFORMATION IS LATCHED AND USED FOR EACHSUCCEEDING VECTOR IN THE ROW  BECAUSE BUFFERING IS USED BETWEEN EACHSTAGE THE COMPUTATIONS MAY BE DONE IN A PIPELINED MANNER  AS AVECTOR PASSES THROUGH A STAGE ANOTHER VECTOR MAY IMMEDIATELY BEPASSED INTO THE STAGE THERE IS NO NEED TO WAIT FOR A SINGLE VECTOR TOPASS ALL THE WAY THROUGH  IT IS THE PIPELINED NATURE OF THEARCHITECTURE THAT LEADS TO ITS EFFICIENCYWHEN USING CORDIC FOR THE QR ALGORITHM SEVERAL ROWS MUST BE MODIFIEDIN SUCCESSION  THIS MAY BE ACCOMPLISHED BY CASCADING SEVERALPIPELINED CORDIC STRUCTURES IN SUCH A WAY THAT A MODIFIED ROW FROM ONESTAGE IS PASSED ON TO THE NEXT STAGE  THIS ALLOWS FOR MOREPARALLELISM IN THE COMPUTATION  ADDITIONAL DETAILS ARE PROVIDED INCITEPROAKISRADER AND CITERADER1996SUBSECTIONRECURSIVE UPDATES TO THE QR FACTORIZATIONLABELSECQRUPDATECONSIDER AGAIN THE LEASTSQUARES FILTERING PROBLEM OFREFEQQRLSFILT1 ONLY NOW CONSIDER THE PROBLEM OF UPDATING THEESTIMATE  THAT IS SUPPOSE THAT DATA QBF1QBF2LDOTSQBFKARE USED TO FORM AN ESTIMATE HBFK BY THE QR METHOD R1K HBFK  QK DBFKNOW A NEW DATA POINT BECOMES AVAILABLE AND WE DESIRE TO COMPUTEHBFK1 USING AS MUCH OF THE PREVIOUS WORK AS POSSIBLEIN THIS CASE IT IS MOST CONVENIENT TO REORDER THE DATA FROMLASTTOFIRST SO WE WILL LET AK  BEGINBMATRIX QBFKH  QBFK1H  VDOTS   QBF1 ENDBMATRIX YBFK   BEGINBMATRIX YK  YK1  CDOTS   Y1ENDBMATRIXTAND DBFK SIMILARLY  AS BEFORE LET  XK  QKRK  QK BEGINBMATRIX R1K  ZEROBFENDBMATRIX WHEN THE NEW DATA COMES THE A MATRIX IS UPDATED AS AK1  BEGINBMATRIX QBFK1H  AK ENDBMATRIXOBSERVE THAT  BEGINBMATRIX1    QHK  ENDBMATRIX AK1  BEGINBMATRIX QBFK1H  RKENDBMATRIX  BEGINBMATRIX QBFK1H  R1K   ZEROBFENDBMATRIX  HTHE MATRIX H HAS THE PROPERTY THAT HIJ  0 FOR I  J1  SUCHA MATRIX IS KNOWN AS AN UPPER EM HESSENBURG INDEXHESSENBURG  MATRIX MATRIX  BY FORCING A ZERO DOWN THE SUBDIAGONAL ELEMENTS OFH IT CAN BE CONVERTED TO AN UPPER TRIANGULAR MATRIX  THIS CAN BEACCOMPLISHED USING A SERIES OF GIVENS ROTATIONS ONE FOR EACHSUBDIAGONAL ELEMENT  LET THE GIVENS ROTATIONS BE INDICATED AS J1J2 LDOTS JM  WE THUS OBTAIN J1 J2 CDOTS JM BEGINBMATRIX1  QHK ENDBMATRIX A RK1  BEGINBMATRIXR1K1  ZEROBF ENDBMATRIXFROM WHICH QK1 CAN ALSO BE IDENTIFIEDBEGINEXERCISESITEM PROVE LEMMA REFLEMQRSAMENORMITEM SHOW THAT FOR A UNITARY MATRIX Q  INDEXUNITARYDETERMINANT BOXEDDETQ  1ITEM COLUMNSPACE PROJECTORS  LET X BE A RANKR MATRIX  SHOW  THAT THE MATRIX WHICH PROJECTS ORTHOGONALLY ONTO THE COLUMN SPACE OF  X IS PX  QMCSUBRANGE1RQMCSUBRANGE1RHWHERE X  QR IS THE QR FACTORIZATION OF X ANDQMCSUBRANGE1R IS THE SC MATLAB NOTATION FOR THE FIRSTR COLUMNS OF QITEM REGARDING THE CAYLEY FORMULA INDEXCAYLEY TRANSFORMATION  BEGINENUMERATE  ITEM SHOW THAT Z IN REFEQCAYLEYPRE HAS Z21  ITEM SHOW THAT U IN REFEQCAYLEYFORM SATISFIES UUHI  ITEM SOLVE REFEQCAYLEYFORM FOR R THUS FINDING A MAPPING    FROM UNITARY MATRICES TO HERMITIAN MATRICES  ITEM A MATRIX S IS EM SKEWSYMMETRIC IF ST  S    INDEXSKEWSYMMETRIC SHOW THAT    IF S IS SKEW SYMMETRIC THEN Q  ISIS1 IS ORTHOGONAL  ENDENUMERATEITEM LABELEXHOUSE1 FOR THE HOUSEHOLDER TRANSFORMATION  REFLECTION MATRIX H  I  2 VBF VBFHVBFHVBF VERIFY THE  FOLLOWING PROPERTIES AND PROVIDE A GEOMETRIC INTERPRETATION  BEGINENUMERATE  ITEM HVBF  VBF  ITEM IF ZBF PERP VBF THEN H ZBF  ZBF  ITEM HH H  HHH  I  ITEM HH  H  ITEM FOR VECTORS XBF AND YBF LA XBFYBF RA  LA HXBFHYBFRATHUS  XBF2   HXBF2  ENDENUMERATEITEM LABELEXSTACKORTHOG SHOW THAT IF Q IS AN ORTHOGONAL MATRIX THEN BEGINBMATRIX 1  ZEROBF  0  Q ENDBMATRIXIS ALSO ORTHOGONALITEM CITEPAGE 74GVL SHOW THAT IF Q  Q1  JQ2 IS UNITARY  WHERE QI IN RBBMATSIZEMM THEN THE MATSIZE2N2N  MATRIX Z  BEGINBMATRIX Q1  Q2  Q2  Q1 ENDBMATRIXIS ORTHOGONALITEM THE HOUSEHOLDER MATRIX DEFINED IN REFEQHVDEF USES A  REFLECTION WITH RESPECT TO AN ORTHOGONAL PROJECTION  IN THIS  PROBLEM WE WILL EXPLORE THE HOUSEHOLDER MATRIX USING A WEIGHTED  PROJECTION AND ITS ASSOCIATED INNER PRODUCT  LET W BE A HERMITIAN  MATRIX AND DEFINE SEE REFEQPRO2MAT2 HVW  I  2PVW  I  2FRACVBF VBFH WVBFH W VBFBEGINENUMERATEITEM SHOW THAT HVWXBFW  XBFW AND THAT  HVWH W HVW  WITEM SHOW THAT HV VBF VBFITEM SHOW THAT HVW HVW  I SO HV IS A REFLECTIONITEM DETERMINE A MEANS OF CHOOSING VBF SO THAT HVW XBF  BEGINBMATRIX1  0  VDOTS  0 ENDBMATRIXALPHAFOR SOME ALPHAENDENUMERATEITEM LABELEQHOUSEMAX CONSIDER THE PROBLEM YBFT  XBFT A  BEGINENUMERATE  ITEM DETERMINE XBF SUCH THAT THE FIRST COMPONENT OF YBF IS    MAXIMIZED SUBJECT TO THE CONSTRAINT THAT  XBF2  1  WHAT    IS THE MAXIMUM VALUE OF Y1 IN THIS CASE  ITEM LET H BE A HOUSEHOLDER MATRIX OPERATING ON THE FIRST COLUMN    OF A  COMMENT ON THE NONZERO VALUE OF HA COMPARED WITH     Y1 OBTAINED IN THE PREVIOUS PART  ENDENUMERATEITEM THE COMPUTATION IN REFEQHOUSELEFT APPLIES THE HOUSEHOLDER  MATRIX TO THE EM LEFT OF A MATRIX AS HV A  DEVELOP AN  EFFICIENT MEANS SUCH AS IN REFEQHOUSELEFT FOR COMPUTING  AHV WITH THE HOUSEHOLDER MATRIX ON THE RIGHTITEM IN THIS PROBLEM YOU WILL DEMONSTRATE THAT DIRECT COMPUTATION OF  THE PSEUDOINVERSE DIRECTLY IS NUMERICALLY INFERIOR TO COMPUTATION  USING A MATRIX FACTORIZATION SUCH AS THE QR OR CHOLESKY  FACTORIZATION  SUPPOSE IT IS DESIRED FIND THE LEASTSQUARES  SOLUTION TO A  BEGINBMATRIX  10000       1000110001   10002 10002   10003 10003   10004 10004   10005 ENDBMATRIXXBF  BEGINBMATRIX20001  20003  20005  20007   20009ENDBMATRIXTHE EXACT SOLUTION IS XBF  11TBEGINENUMERATEITEM DETERMINE THE CONDITION NUMBER OF A AND ATA IF POSSIBLEITEM COMPUTE THE LEASTSQUARES SOLUTION USING THE FORMULA  XBFHAT  ATA1AT BBF EXPLICITLYITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR  DECOMPOSITION IS COMPUTED USING HOUSEHOLDER TRANSFORMATIONSITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR  DECOMPOSITION IS COMPUTED USING GRAMSCHMIDTITEM COMPUTE THE SOLUTION USING THE CHOLESKY FACTORIZATIONITEM COMPARE THE ANSWERS AND COMMENTENDENUMERATEITEM ROTATION MATRICES INDEXROTATION MATRIX VERIFY THE FOLLOWING  PROPERTIES ABOUT ROTATION MATRICES OF THE FORM AND PROVIDE A  GEOMETRIC INTERPRETATION GTHETA  BEGINBMATRIX COS THETA  SIN THETA  SIN  THETA  COS THETA ENDBMATRIXBEGINENUMERATEITEM GTHETA GTHETA  IITEM GTHETA2  G2THETAITEM GTHETA GPHI  GTHETAPHIENDENUMERATETHUS GTHETA THETA IN RBB FORMS A GROUP  ITEM CITERADER1996 THE GRAMMIAN MATRIX FOR A LEASTSQUARES PROBLEM IS RK  AHAWHERE A  BEGINBMATRIXQBF1H  QBF2H  VDOTS  QBFKHENDBMATRIXLET Z  AH SO WE CAN WRITE R  ZZHBEGINENUMERATEITEM SHOW THAT IF Z1   Z Q1 WHERE Q1 IS A UNITARY MATRIX  THEN WE CAN WRITE R  Z1 Z1HITEM DESCRIBE HOW TO CONVERT Z TO A LOWER TRIANGULAR MATRIX L BY A  SERIES OF ORTHOGONAL TRANSFORMATIONS  THUS WE CAN WRITE R  LLHITEM DESCRIBE HOW TO SOLVE THE EQUATION R XBF  YBF FOR YBF  BASED UPON THIS REPRESENTATION OF R  ENDENUMERATENOTE THAT SINCE WE NEVER HAVE TO COMPUTE R EXPLICITLY THE NUMERICALPROBLEMS OF COMPUTING AHA NEVER ARISE  FOR FIXED POINT ARITHMETICTHE WORDLENGTH REQUIREMENTS ARE APPROXIMATELY HALF AS LONG ASCOMPUTING THE GRAMMIAN AND THEN FACTORING IT CITERADERSTEINHARDTITEM THE QR FACTORIZATION USING GIVENS ROTATIONS WAS GIVEN ONLY FOR  EM REAL MATRICES  DETERMINE A MODIFICATION TO THE ALGORITHM TO  HANDLE COMPLEX MATRICES  HINT ROTATE IN PHASEITEM WRITE AND TEST AN EFFICIENT ROUTINE TO COMPUTE QT B USING THE TT    THETA DATA RETURNED FROM TT QRGIVENS SIMILAR TO THE ROUTINE  TT QRTBT WRITTEN FOR THE HOUSEHOLDER METHOD  AN APPROPRIATE  CALLING SEQUENCE MIGHT BE TT QRTBTGIVBTHETAMN  QRQTBGIVMITEM IN THE CORDIC ALGORITHM THE MICROROTATION ANGLES ARE THETAI   TAN1 2I I01LDOTS  PROVE THAT IF THETA LEQ  THETAK THEN A REPRESENTATION OF THETA EXISTS OF THE FORM THETA  SUMIK1INFTY RHOI THETAIITEM DETERMINE A REPRESENTATION OF THETA  23CIRC USING THE  ANGLES IN THE CORDIC REPRESENTATIONITEM FOR THE LU FACTORIZATION IT IS POSSIBLE TO REPRESENT BOTH THE  L AND U FACTORS IN THE ORIGINAL MATRIX A WITH POSSIBLY SOME  PERMUTATION INFORMATION STORED SEPARATELY  DETERMINE A MEANS BY  WHICH Q AND R FACTORS CAN BE STORED IN THE ORIGINAL MATRIX A  FOR BOTH THE HOUSEHOLDER AND GIVENS METHODSITEM ONE WAY TO GENERALIZE THE GIVENS METHOD IS TO DEAL WITH A  ROTATION MATRIX WITH COMPLEX NUMBERS  FOR A VECTOR XBF IN  CBB2 FIND AN ALGORITHM FOR DETERMINING A UNITARY MATRIX OF THE  FORM  Q  BEGINBMATRIX C  SBAR  SC ENDBMATRIX SUCH THAT C IN RBB C2  S2 1 AND THE SECOND COMPONENT OF QXBF IS ZEROITEM SUPPOSE A  I  VBF VBFT  FIND THE CHOLESKY FACTORIZATION  OF AITEM FAST GIVENS TRANSFORMATIONS  LET D BE A DIAGONAL MATRIX LET  M BE A MATRIX SUCH THAT MH M  D AND LET Q  MD12  BEGINENUMERATE  ITEM SHOW THAT Q IS ORTHOGONAL  ITEM FOR A MATSIZE22 MATRIX M OF THE FORM M  BEGINBMATRIX BETA  1  1  ALPHA ENDBMATRIXSHOW HOW TO CHOOSE ALPHA AND BETA SO THAT FOR A 2VECTOR XBF M XBF  BEGINBMATRIX TIMES  0 ENDBMATRIXTHAT IS M SETS THE SECOND COMPONENT OF XBF TO ZERO AND MDMH  D1IS DIAGONAL  THUS MXBF ACTS LIKE A GIVENS ROTATION BUT WITHOUTTHE NEED TO COMPUTE A SQUARE ROOTITEM DESCRIBE HOW TO APPLY THE MATSIZE22 MATRIX TO PERFORM A  FAST QR DECOMPOSITION OF A MATRIX A ITEM WRITE AND TEST A SC MATLAB FUNCTION TO IMPLEMENT THE FAST QR  ENDENUMERATEFURTHER INFORMATION ON FAST GIVENS INCLUDING SOME IMPORTANT ISSUES OFSTABILIZING THE NUMERICAL COMPUTATIONS ARE GIVEN IN CITEGVLITEM LABELEXPOTH IN THE TEXT RELATED TO FIGURE REFFIGHOUSE1  IT WAS STATED THAT THE DIAGONALS OF A PARALLELOGRAM ARE ORTHOGONAL  PROVE THAT THIS IS TRUEITEM MATRIX SPACES FROM THE QR FACTORIZATION  INDEXFOUR    FUNDAMENTAL SUBSPACESIF A IN MMN  WHERE MN AND A HAS FULL COLUMN RANK THE QR FACTORIZATION CAN  BE WRITTEN AS A  Q1 Q2 BEGINBMATRIXR1  0 ENDBMATRIXWHERE Q1 IN MMN AND Q2 IN MMMN AND R1 INMNN  SHOW THATBEGINENUMERATEITEM AQ1R1  THIS IS KNOWN AS THE SKINNY QR FACTORIZATION   OBSERVE THAT THE COLUMNS OF Q1 ARE ORTHOGONALITEM RANGEA  RANGEQ1ITEM RANGEAPERP  RANGEQ2ENDENUMERATEENDEXERCISESSETEXSECTREFSECLUFACTBEGINEXERCISESITEM FOR THE MATRIX A  BEGINBMATRIX 2  5  9  1  4  7  3  2 1ENDBMATRIXDETERMINE THE LU FACTORIZATION BOTH WITH AND WITHOUT PIVOTING ITEM SHOW HOW TO OBTAIN THE L MATRIX FROM THE TT LU RETURNED   FROM TT NEWLUITEM WRITE A TT MATLAB ROUTINE TO SOLVE THE SYSTEM OF EQUATIONS  AXBF  BBF ASSUMING THAT THE LU FACTORIZATION IS OBTAINED USING  TT NEWLUITEM VERIFY THE FOLLOWING FACTS ABOUT TRIANGULAR MATRICES  BEGINENUMERATE  ITEM THE INVERSE OF AN UPPER TRIANGULAR MATRIX IS UPPER    TRIANGULAR  THE INVERSE OF A LOWER TRIANGULAR MATRIX IS LOWER    TRIANGULAR  ITEM THE PRODUCT OF TWO UPPER TRIANGULAR MATRICES IS UPPER TRIANGULAR  ENDENUMERATE ITEM SHOW THAT IF A MATRIX IS DIAGONALLY DOMINANT THEN NO PIVOTING   IS REQUIRED TO ENSURE THAT LIJ  1ITEM LABELEXNUMPOOR THIS EXERCISE ILLUSTRATES THE POTENTIAL  DIFFICULTY OF LU FACTORIZATION WITHOUT PIVOTING  SUPPOSE IT IS  DESIRED TO SOLVE THE SYSTEM OF EQUATIONS BEGINBMATRIX 2 45  612001  1  4  8  3ENDBMATRIXXBF  BEGINBMATRIX 5 33002 21ENDBMATRIXTHE TRUE SOLUTION TO THIS SYSTEM OF EQUATIONS IS XBF  1 2 3TAND THE MATRIX A IS VERY WELL CONDITIONED  COMPUTE THE SOLUTION TOTHIS PROBLEM USING THE LU DECOMPOSITION WITHOUT PIVOTING USINGARITHMETIC ROUNDED TO THREE SIGNIFICANT PLACES  THEN COMPUTE USINGPIVOTING AND COMPARE THE ANSWERS WITH THE EXACT RESULTEXSKIP SETEXSECTREFSECCHOLESKYITEM COMPUTE THE CHOLESKY FACTORIZATION OF  A  BEGINBMATRIX 464  62518  41822 ENDBMATRIXAS ALLT  THEN WRITE THIS AS A  UTDU WHERE U IS AN UPPER TRIANGULAR MATRIXWITH 1 ALONG THE DIAGONALITEM SHOW THAT REFEQBCHOL IS TRUEITEM GIVEN A ZEROMEAN DISCRETETIME INPUT SIGNAL FT WHICH WE  FORM INTO A VECTOR QBFT  BEGINBMATRIX FBART  FBART1  CDOTS   FBARTMENDBMATRIXTWE DESIRE TO FORM A SET OF OUTPUTS BBFT  BEGINBMATRIX B0T  B1T  CDOTS  BMTENDBMATRIXTBY BBFT  HQBFT THAT ARE UNCORRELATED THAT IS EBIT BBARJT  0 TEXT IF INEQ JLET R  EQBFT QBFHT BE THE CORRELATION MATRIX OF THE INPUTDATA  DETERMINE THE MATRIX H WHICH DECORRELATES THE INPUT DATA  ITEM LET X  XBF1 XBF2 LDOTS XBFN BE A SET OF    REALVALUED ZEROMEAN DATA WITH CORRELATION MATRIX RXX  FRAC1N XXTDETERMINE A TRANSFORMATION ON X THAT PRODUCES A DATA SET Y  Y  H XSUCH THAT RYY  FRAC1N YYTIS EQUAL TO AN IDENTITY ITEM WRITE A SC MATLAB ROUTINE TT BACKCHOL WHICH FACTORS A   SYMMETRIC POSITIVE DEFINITE MATRIX AS A  UUH WHERE U IS AN   UPPER TRIANGULAR MATRIXITEM WRITE SC MATLAB ROUTINES TT X FORSUBLB AND TT    BACKSUBUB TO SOLVE LXBF  BBF FOR A LOWER TRIANGULAR MATRIX    L AND U XBF  BBF FOR AN UPPER TRIANGULAR MATRIX U  ITEM DEVELOP A MEANS OF COMPUTING THE SOLUTION TO THE WEIGHTED    LEASTSQUARES PROBLEM XBF  AHWA1 AH W BBF USING THE    CHOLESKY FACTORIZATION ITEM SUPPOSE A  I  VBF VBFT  FIND THE CHOLESKY FACTORIZATION   OF AEXSKIPSETEXSECTREFSECQRCOMP ITEM PROVE LEMMA REFLEMQRSAMENORMITEM SHOW THAT FOR A UNITARY MATRIX Q  INDEXUNITARYDETERMINANT BOXEDDETQ  1ITEM COLUMNSPACE PROJECTORS  LET X BE A RANKR MATRIX  SHOW  THAT THE MATRIX WHICH PROJECTS ORTHOGONALLY ONTO THE COLUMN SPACE OF  X IS PX  QMCSUBRANGE1RQMCSUBRANGE1RHWHERE X  QR IS THE QR FACTORIZATION OF X ANDQMCSUBRANGE1R IS THE SC MATLAB NOTATION FOR THE FIRSTR COLUMNS OF QITEM REGARDING THE CAYLEY FORMULA INDEXCAYLEY TRANSFORMATION  BEGINENUMERATE  ITEM SHOW THAT Z IN REFEQCAYLEYPRE HAS Z21  ITEM SHOW THAT U IN REFEQCAYLEYFORM SATISFIES UUHI  ITEM SOLVE REFEQCAYLEYFORM FOR R THUS FINDING A MAPPING    FROM UNITARY MATRICES TO HERMITIAN MATRICES  ITEM A MATRIX S IS EM SKEWSYMMETRIC IF ST  S    INDEXSKEWSYMMETRIC SHOW THAT    IF S IS SKEW SYMMETRIC THEN Q  ISIS1 IS ORTHOGONAL  ENDENUMERATEITEM LABELEXHOUSE1 FOR THE HOUSEHOLDER TRANSFORMATION  REFLECTION MATRIX H  I  2 VBF VBFHVBFHVBF VERIFY THE  FOLLOWING PROPERTIES AND PROVIDE A GEOMETRIC INTERPRETATION  BEGINENUMERATE  ITEM HVBF  VBF  ITEM IF ZBF PERP VBF THEN H ZBF  ZBF  ITEM HH  H  ITEM HH H  HHH  I  ITEM FOR VECTORS XBF AND YBF LA XBFYBF RA  LA HXBFHYBFRATHUS  XBF2   HXBF2  ENDENUMERATEITEM DETERMINE A ROTATION THETA IN C  COS THETA AND S   SIN THETA SUCH THAT BEGINBMATRIX C  S  S  C ENDBMATRIXBEGINBMATRIX3  4ENDBMATRIX  BEGINBMATRIX5  0 ENDBMATRIXITEM LABELEXSTACKORTHOG SHOW THAT IF Q IS AN ORTHOGONAL MATRIX THEN BEGINBMATRIX 1  ZEROBF  0  Q ENDBMATRIXIS ALSO ORTHOGONALITEM CITEPAGE 74GVL SHOW THAT IF Q  Q1  JQ2 IS UNITARY  WHERE QI IN RBBMATSIZEMM THEN THE MATSIZE2M2M  MATRIX Z  BEGINBMATRIX Q1  Q2  Q2  Q1 ENDBMATRIXIS ORTHOGONALITEM THE HOUSEHOLDER MATRIX DEFINED IN REFEQHVDEF USES A  REFLECTION WITH RESPECT TO AN ORTHOGONAL PROJECTION  IN THIS  PROBLEM WE WILL EXPLORE THE HOUSEHOLDER MATRIX USING A WEIGHTED  PROJECTION AND ITS ASSOCIATED INNER PRODUCT  LET W BE A HERMITIAN  MATRIX AND DEFINE SEE REFEQPROJMAT2 HVW  I  2PVW  I  2FRACVBF VBFH WVBFH W VBFBEGINENUMERATEITEM SHOW THAT HVWH W HVW  W AND THAT HVWXBFW   XBFWWHERE XBFW  XBFH W XBFITEM SHOW THAT HVW VBF VBFITEM SHOW THAT HVW HVW  I SO HVW IS A REFLECTIONITEM DETERMINE A MEANS OF CHOOSING VBF SO THAT HVW XBF  BEGINBMATRIX1  0  VDOTS  0 ENDBMATRIXALPHAFOR SOME ALPHAENDENUMERATEITEM LABELEXHOUSEMAX CONSIDER THE PROBLEM YBFT  XBFT A  BEGINENUMERATE  ITEM DETERMINE XBF SUCH THAT THE FIRST COMPONENT OF YBF IS    MAXIMIZED SUBJECT TO THE CONSTRAINT THAT  XBF2  1  WHAT    IS THE MAXIMUM VALUE OF Y1 IN THIS CASE  ITEM LET H BE A HOUSEHOLDER MATRIX OPERATING ON THE FIRST COLUMN    OF A  COMMENT ON THE NONZERO VALUE OF THE FIRST COLUMN HA    COMPARED WITH Y1 OBTAINED IN THE PREVIOUS PART  ENDENUMERATEITEM LET XBF AND YBF BE NONZERO VECTORS IN RBBN  DETERMINE AHOUSEHOLDER MATRIX P SUCH THAT PXBF IS A MULTIPLE OF YBFGIVE A GEOMETRIC INTERPRETATION OF YOUR ANSWERITEM THE COMPUTATION IN REFEQHOUSELEFT APPLIES THE HOUSEHOLDER  MATRIX TO THE EM LEFT OF A MATRIX AS HV A  DEVELOP AN  EFFICIENT MEANS SUCH AS IN REFEQHOUSELEFT FOR COMPUTING  AHV WITH THE HOUSEHOLDER MATRIX ON THE RIGHTITEM IN THIS PROBLEM YOU WILL DEMONSTRATE THAT DIRECT COMPUTATION OF  THE PSEUDOINVERSE DIRECTLY IS NUMERICALLY INFERIOR TO COMPUTATION  USING A MATRIX FACTORIZATION SUCH AS THE QR OR CHOLESKY  FACTORIZATION  SUPPOSE IT IS DESIRED FIND THE LEASTSQUARES  SOLUTION TO A  BEGINBMATRIX  10000       1000110001   10002 10002   10003 10003   10004 10004   10005 ENDBMATRIXXBF  BEGINBMATRIX20001  20003  20005  20007   20009ENDBMATRIXTHE EXACT SOLUTION IS XBF  11TBEGINENUMERATEITEM DETERMINE THE CONDITION NUMBER OF A AND ATA IF POSSIBLEITEM COMPUTE THE LEASTSQUARES SOLUTION USING THE FORMULA  XBFHAT  ATA1AT BBF EXPLICITLYITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR  DECOMPOSITION IS COMPUTED USING HOUSEHOLDER TRANSFORMATIONS ITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR   DECOMPOSITION IS COMPUTED USING GRAMSCHMIDTITEM COMPUTE THE SOLUTION USING THE CHOLESKY FACTORIZATIONITEM COMPARE THE ANSWERS AND COMMENTENDENUMERATEITEM ROTATION MATRICES INDEXROTATION MATRIX VERIFY THE FOLLOWING  PROPERTIES ABOUT ROTATION MATRICES OF THE FORM AND PROVIDE A  GEOMETRIC INTERPRETATION GTHETA  BEGINBMATRIX COS THETA  SIN THETA  SIN  THETA  COS THETA ENDBMATRIXBEGINENUMERATEITEM GTHETA GTHETA  IITEM GTHETA GPHI  GTHETAPHIENDENUMERATENOTE THAT GTHETA THETA IN RBB FORMS A GROUP  ITEM CITERADER1996 THE GRAMMIAN MATRIX FOR A LEASTSQUARES PROBLEM IS RK  AHAWHERE A  BEGINBMATRIXQBF1H  QBF2H  VDOTS  QBFKHENDBMATRIXLET Z  AH SO WE CAN WRITE RK  ZZHBEGINENUMERATEITEM SHOW THAT IF Z1   Z Q1 WHERE Q1 IS A UNITARY MATRIX  THEN WE CAN WRITE RK  Z1 Z1HITEM DESCRIBE HOW TO CONVERT Z TO A LOWER TRIANGULAR MATRIX L BY A  SERIES OF ORTHOGONAL TRANSFORMATIONS  THUS WE CAN WRITE RK  LLHITEM DESCRIBE HOW TO SOLVE THE EQUATION RK XBF  YBF FOR YBF  BASED UPON THIS REPRESENTATION OF RK  ENDENUMERATENOTE THAT SINCE WE NEVER HAVE TO COMPUTE R EXPLICITLY THE NUMERICALPROBLEMS OF COMPUTING AHA NEVER ARISE  FOR FIXED POINT ARITHMETICTHE WORDLENGTH REQUIREMENTS ARE APPROXIMATELY HALF AS LONG ASCOMPUTING THE GRAMMIAN AND THEN FACTORING IT CITERADERSTEINHARDT ITEM THE QR FACTORIZATION USING GIVENS ROTATIONS WAS GIVEN ONLY FOR   EM REAL MATRICES  DETERMINE A MODIFICATION TO THE ALGORITHM TO   HANDLE COMPLEX MATRICES  HINT ROTATE IN PHASEITEM WRITE AND TEST AN EFFICIENT ROUTINE TO COMPUTE QT B USING THE TT    THETA DATA RETURNED FROM TT QRGIVENS SIMILAR TO THE ROUTINE  TT QRTBT WRITTEN FOR THE HOUSEHOLDER METHOD  AN APPROPRIATE  CALLING SEQUENCE MIGHT BE TT QRTBTGIVBTHETAMN  QRQTBGIVM ITEM IN THE CORDIC ALGORITHM THE MICROROTATION ANGLES ARE THETAI    TAN1 2I I01LDOTS  PROVE THAT IF THETA LEQ   THETAK THEN A REPRESENTATION OF THETA EXISTS OF THE FORM  THETA  SUMIK1INFTY RHOI THETAI ITEM DETERMINE A REPRESENTATION OF THETA  23CIRC USING THE  ANGLES IN THE CORDIC REPRESENTATIONITEM WE HAVE SEEN THAT IN THE LU FACTORIZATION IT IS POSSIBLE TO OVERWRITETHE ORIGINAL MATRIX A WITH INFORMATION ABOUT THE L AND UFACTORS WITH POSSIBLY SOME PERMUTATION INFORMATION STOREDSEPARATELY  IN THIS QUESTION WE WILL DETERMINE THAT THE SAMEOVERWRITING REPRESENTATION OF A ALSO WORKS FOR HOUSEHOLDER ANDGIVENS APPROACHES TO THE QR FACTORIZATION  BEGINENUMERATEITEM DETERMINE A MEANS BY WHICH THE Q AND R FACTORS COMPUTED  USING HOUSEHOLDER TRANSFORMATIONS CAN BE OVERWRITTEN IN THE ORIGINAL  A MATRIX  HINT LET VBF1  1ITEM DETERMINE HOW THE Q AND R FACTORS COMPUTED USING GIVENS  TRANSFORMATIONS CAN BE OVERWRITTEN IN THE ORIGINAL A MATRIXENDENUMERATE ITEM ONE WAY TO GENERALIZE THE GIVENS METHOD IS TO DEAL WITH A   ROTATION MATRIX WITH COMPLEX NUMBERS  FOR A VECTOR XBF IN   CBB2 FIND AN ALGORITHM FOR DETERMINING A UNITARY MATRIX OF THE   FORM   Q  BEGINBMATRIX C  SBAR  SC ENDBMATRIX   SUCH THAT C IN RBB C2  S2 1 AND THE SECOND COMPONENT OF Q XBF IS ZEROITEM FAST GIVENS TRANSFORMATIONSINDEXGIVENS TRANSFORMATIONSFAST  LET D BE A DIAGONAL MATRIX LET  M BE A MATRIX SUCH THAT MT M  D AND LET Q  MD12  BEGINENUMERATE  ITEM SHOW THAT Q IS ORTHOGONAL  ITEM FOR A MATSIZE22 MATRIX M1 OF THE FORM M1  BEGINBMATRIX BETA  1  1  ALPHA ENDBMATRIXSHOW HOW TO CHOOSE ALPHA AND BETA SO THAT FOR A 2VECTOR XBF M1 XBF  BEGINBMATRIX TIMES  0 ENDBMATRIXTHAT IS M SETS THE SECOND COMPONENT OF XBF TO ZERO AND M1DM1H  D1IS DIAGONAL  THUS M1XBF ACTS LIKE A GIVENS ROTATION BUT WITHOUTTHE NEED TO COMPUTE A SQUARE ROOTITEM DESCRIBE HOW TO APPLY THE MATSIZE22 MATRIX TO PERFORM A  FAST QR DECOMPOSITION OF A MATRIX A  ITEM WRITE AND TEST A SC MATLAB FUNCTION TO IMPLEMENT THE FAST QRENDENUMERATEFURTHER INFORMATION ON FAST GIVENS INCLUDING SOME IMPORTANT ISSUES OFSTABILIZING THE NUMERICAL COMPUTATIONS ARE GIVEN IN CITEGVLITEM LABELEXPOTH IN THE TEXT RELATED TO FIGURE REFFIGHOUSE1  IT WAS STATED THAT THE DIAGONALS OF AN EQUILATERAL PARALLELOGRAM ARE ORTHOGONAL  PROVE THAT THIS IS TRUEITEM MATRIX SPACES FROM THE QR FACTORIZATION  INDEXFOUR FUNDAMENTAL SUBSPACESIF A IN MMN  WHERE MN AND A HAS FULL COLUMN RANK THE QR FACTORIZATION CAN  BE WRITTEN AS A  Q1 Q2 BEGINBMATRIXR1  0 ENDBMATRIXWHERE Q1 IN MMN AND Q2 IN MMMN AND R1 INMNN  SHOW THATBEGINENUMERATEITEM AQ1R1  THIS IS KNOWN AS THE SKINNY QR FACTORIZATION   OBSERVE THAT THE COLUMNS OF Q1 ARE ORTHOGONALITEM RANGEA  RANGEQ1ITEM RANGEAPERP  RANGEQ2ENDENUMERATEENDEXERCISESSECTIONREFERENCESLABELSECREFFACTCOMPUTATION OF MATRIX FACTORIZATIONS IS WIDELY DISCUSSED IN A VARIETYOF NUMERICAL ANALYSIS TEXTS  THE CONNECTION OF THE LU WITH GAUSSIANELIMINATION IS DESCRIBED WELL IN CITESTRANG1988  MOST OF THEMATERIAL HERE ON THE QR FACTORIZATION HAS BEEN DRAWN FROMCITEGVL  IN ADDITION TO FACTORIZATIONS THIS SOURCE ALSO PROVIDESPERTURBATION ANALYSES OF THE ALGORITHMS AND COMPARISONS OF VARIANTSOF THE ALGORITHMS  A FAST GIVENS ROTATION ALGORITHM WHICH DOESNOT REQUIRE SQUARE ROOTS IS ALSO PRESENTED THERE  VARIANTS ON THECHOLESKY ALGORITHM PRESENTED HERE ARE PRESENTED IN CITEGVL  UPDATEALGORITHMS FOR THE QR FACTORIZATION IN ADDITION TO THE ONE FOR UPDATEBY ADDING A ROW ARE PRESENTED INCLUDING UPDATES FOR A RANKONEMODIFICATION AND COLUMN MODIFICATIONS ARE ALSO PRESENTED INCITEGVLTHE HOUSEHOLDER TRANSFORMATION APPEARED IN CITEHOUSEHOLDER1958APPLICATION OF HOUSEHOLDER TRANSFORMATIONS WITH WEIGHTED PROJECTIONSIS DISCUSSED IN CITERADERSTEINHARDTAPPLICATION OF QR FACTORIZATIONS TO LEASTSQUARES FILTERING ISEXTENSIVELY DISCUSSED IN CITEPROAKISRADER AND CITEHAYKIN1996CITEPROAKISRADER ALSO DEMONSTRATES APPLICATION OF GRAMSCHMIDT  ANDMODIFIED GRAMSCHMIDT TO LEASTSQUARES AND RECURSIVE UPDATES OFLEASTSQUARES  A DISCUSSION OF APPLICATIONS OF HOUSEHOLDER TRANSFORMSTO SIGNAL PROCESSING APPEARS IN CITESTEINHARDT1988 LOCAL VARIABLES TEXMASTER TEST ENDSECTIONOPERATOR NORMSLABELSECMATNORMAN OPERATOR NORM LIKE ANY NORM MUST SATISFY THE PROPERTIES DESCRIBEDIN SECTION REFSECNORMVS  THERE ARE SEVERAL DIFFERENT WAYS OFDEFINING THE NORM OF A TRANSFORMATION OPERATOR  ONE WAY IS TODEFINE THE NORM SO THAT IT PROVIDES INDICATION OF THE MAXIMAL AMOUNTOF CHANGE OF LENGTH OF A VECTOR THAT IT OPERATES ON  LET X AND YBE LP OR LP AND LET A BE A LINEAR OPERATOR AMC XRIGHTARROWY  THE P BF OPERATOR NORM OR PNORM OR LP NORM OF AIS  A P  SUPXIN X NEQ 0FRACAX PXP SUPX IN X X P  1AXPWHERE  CDOT P IS THE PNORM DEFINED IN SECTIONREFSECNORMVS  NOTE AX IN Y SO THE NORM AXP IS THENORM ON Y  WE COULD IN GENERAL USE DIFFERENT NORMS FOR X ANDAX BUT USUALLY THIS IS NOT DONE  THE NORM ON A SO OBTAINEDIS SAID TO BE EM SUBORDINATE TO THE NORM ON XINDEXSUBORDINATE NORM INDEXNORMSUBORDINATE FOR A SUBORDINATENORM IT IS STRAIGHTFORWARD TO VERIFY THAT I  1 WHERE I ISTHE IDENTITY OPERATOR  GEOMETRICALLY A SUBORDINATE NORM MEASURES THEMAXIMUM EXTENT THAT A TRANSFORMS THE UNIT CIRCLE  THE CONCEPT ISSHOWN IN FIGURE REFFIGOPNORMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRNORM1    CAPTIONGEOMETRY OF THE OPERATOR NORM    LABELFIGOPNORM  ENDCENTERENDFIGURETHE PNORMS HAVE THE PROPERTY THAT AXBF P LEQ AP XBFPTHUS A BOUNDS THE AMPLIFYING POWER OF THE MATRIX AALSO THE PNORMS SATISFY THE BF SUBMULTIPLICATIVE PROPERTYINDEXSUBMULTIPLICATIVE PROPERTY ABP LEQ AP B PTHIS IS STRAIGHTFORWARD TO SHOW SINCE BY THE DEFINITION OF THE PNORMFOR ALL X IN X  AB X  LEQ  A   BX  LEQ A B X SUBSECTIONBOUNDED OPERATORSTHIS SECTION IS SOMEWHAT TECHNICAL AND MANY READERS MAY NEED ONLY THEFIRST DEFINITIONBEGINDEFINITIONIF THE NORM OF A TRANSFORMATION IS FINITE THE TRANSFORMATION IS SAIDTO BE EM BOUNDED INDEXBOUNDEDENDDEFINITIONTHE FOLLOWING THEOREM PRESENTS A RATHER REMARKABLE FACT ABOUTBOUNDED LINEAR OPERATORS BEGINTHEOREM LABELTHMCONTBOUND   A LINEAR OPERATOR AMC X RIGHTARROW Y IS BOUNDED IF AND ONLY IF   IT IS CONTINUOUS ENDTHEOREMSINCE A LINEAR FUNCTIONAL IS A LINEAR OPERATOR THE SAME THEOREMAPPLIES TO FUNCTIONALSBEGINPROOF  SUPPOSE THAT A IS BOUNDED WITH M SUCH THAT AX LEQ M  X FOR ALL X IN X  LET XN BE A SEQUENCE APPROACHING  ZERO XN RIGHTARROW 0  THEN AXN LEQ M  XNRIGHTARROW  0  BY THE PROPERTIES OF CONTINUITY CONTINUOUS FUNCTIONS PRESERVE  CONVERGENCE IT FOLLOWS THAT A IS CONTINUOUS    CONVERSELY ASSUME A IS CONTINUOUS  THEN THERE IS A DELTA  0  SUCH THAT AX1 FOR X   DELTA  THEN SINCE THE NORM OF  DELTA XX IS EQUAL TO DELTA A X   A XDELTA X X XDELTA XDELTATHE VALUE M1DELTA SERVES AS A BOUND FOR AENDPROOFTHE FOLLOWING THEOREM IS OF GREAT UTILITY BY SHOWING THAT LINEAROPERATORS FROM FINITEDIMENSIONAL SPACE ARE CONTINUOUS WE CANCONCLUDE FROM THE PREVIOUS THEOREM THAT THEY ARE ALSO BOUNDED  SINCE MANYOF THE RESULTS OF THIS CHAPTER RELY ON BOUNDED LINEAR OPERATORS THISTHEOREM REASSURES US THAT MATRICES  OPERATORS ON FINITE DIMENSIONALSPACES  WILL WORKBEGINTHEOREM LABELTHMFDBD  LET AMC X RIGHTARROW Y BE A LINEAR OPERATOR WHERE X AND Y  ARE NORMED LINEAR SPACES  IF X IS FINITE DIMENSIONAL THEN A IS  CONTINUOUSENDTHEOREMNOTE THAT THIS THEOREM DOES NOT ASSUME THAT Y IS FINITE DIMENSIONALPROOF OF THEOREM REFTHMFDBD MAKES USE OF THE FOLLOWING LEMMAWHICH IS THE MOST TECHNICAL PART OF THIS SECTIONBEGINLEMMA CITEPAGE 265NAYLORSELL LABELLEMLBD  LET X BE A FINITEDIMENSIONAL NORMED LINEAR SPACE AND LET  XBF1XBF2LDOTS XBFN BE A HAMEL BASIS FOR  X INDEXHAMEL BASIS  THEN FOR XBF IN X EACH COEFFICIENT  ALPHAI IN THE EXPANSION XBF  ALPHA1 XBF1  ALPHA2 XBF2  CDOTS  ALPHAN XBFNIS A CONTINUOUS LINEAR FUNCTION OF XBF  BEING CONTINUOUS IT ISBOUNDED SO THERE IS A CONSTANT M SUCH THAT ALPHAI LEQ M XBFENDLEMMABEGINPROOFSHOWING LINEARITY IS STRAIGHTFORWARD AND IS OMITTEDIT WILL SUFFICE TO SHOW THAT THERE IS AN M0 SUCH THATBEGINEQUATION MALPHA1  ALPHA2  CDOTS  ALPHAN LEQ XBFLABELEQLBD1ENDEQUATIONSINCE IT FOLLOWS THAT ALPHAI LEQ M1  XBF  WE WILLPROVE REFEQLBD1 FIRST FOR COEFFICIENTS  ALPHA1LDOTSALPHAN SATISFYING THE CONDITION ALPHA1 CDOTS  ALPHAN 1  LET A  ALPHA1LDOTSALPHAN ALPHA1 CDOTS ALPHAN1THIS SET IS CLOSED AND BOUNDED COMPACT  NOW DEFINE A FUNCTIONFMC A RIGHTARROW RBB BYBEGINEQUATION FALPHA1LDOTSALPHAN   ALPHA1 XBF1  CDOTS ALPHAN XBFNLABELEQLBD2ENDEQUATIONIT CAN BE SHOWN THAT F CONTINUOUS AND IT IS CLEAR THAT F0  LET M  MINALPHA1LDOTSALPHAN IN A FALPHA1LDOTSALPHANSINCE F IS CONTINUOUS ON A CLOSED BOUNDED SET THIS MINIMUM DOESEXIST FOR SOME POINT ALPHA1 LDOTS ALPHAN IN A  HENCEWE HAVE FOUND A POINT M THAT SATISFIES REFEQLBD1  IF M0THEN ALPHA1 XBF1  CDOTS  ALPHAN XBFN  0CONTRADICTING THE FACT THAT XBFI IS A BASIS LINEARLYINDEPENDENT  HENCE M0FOR GENERAL SETS OF COEFFICIENTS  ALPHAI SET BETA ALPHA1  CDOTS  ALPHAN  IF BETA0 THE RESULT ISTRIVIAL  IF BETA0 THEN WE WRITE BEGINALIGNED ALPHA1 XBF1  CDOTS  ALPHAN XBFN  BETAALPHA1BETA XBF1  CDOTS  ALPHANBETA XBFN  BETA FALPHA1BETALDOTSALPHANBETA  GEQ MBETA GEQMALPHA1  CDOTS ALPHANENDALIGNEDENDPROOFBEGINPROOF OF THEOREM REFTHMFDBD  LET XBF1XBF2LDOTSXBFN BE A HAMEL BASIS FOR X  LET XBF  IN X BE EXPRESSED IN TERMS OF THIS BASIS AS XBF  ALPHA1 XBF1  ALPHA2 XBF2  CDOTS  ALPHAN XBFNLET D  MAX1 LEQ I LEQ N A XBFI  THEN BEGINALIGNEDAXBF  AALPHA1 XBF1  ALPHA2 XBF2  CDOTS  ALPHANXBFN LEQ ALPHA1A XBF1  ALPHA2A XBF2 CDOTS  ALPHANA XBFN  LEQ DALPHA1  ALPHA2  CDOTS  ALPHANENDALIGNEDNOW BY THE LEMMA ABOVE THERE IS AN M SUCH THAT ALPHA1  CDOTSALPHAN LEQ M XBF SO THAT  A XBF  LEQ DM  XBFENDPROOFBEFORE CONSIDERING THE IMPORTANT SPECIAL CASE OF MATRIX TRANSFORMATIONSWE WILL CONSIDER SOME MORE GENERALIZED TRANSFORMATIONSBEGINEXAMPLELET X  C01 AND DEFINE AXRIGHTARROW X BY AXT  INT01 KTTAUXTAUDTAUWHERE T IN 01 AND K IS CONTINUOUS  WE WILL COMPUTE THELINFTY NORM OF THIS OPERATORBEGINALIGNED A X  MAXT IN 01 BIGLINT01 KTTAUXTAUDTAUBIGR  LEQ MAXT IN 01 INT01 KTTAUDTAU MAXT IN 01XT   MAXT IN 01 INT01 KTTAUDTAU XENDALIGNEDIT CAN BE SHOWN THAT THE INEQUALITY CAN BE ACHIEVED SO THAT A  MAXT IN 01 INT01 KTTAUDTAUSINCE KTTAU IS CONTINUOUS THEN A IS BOUNDEDENDEXAMPLEBEGINEXAMPLELET AC101 RIGHTARROW C01 BE THE OPERATOR AX  FRACDDTXTHE FUNCTION XT  SIN OMEGA0 T IN C101 HAS UNIFORM NORM 1FOR ANY VALUE OF OMEGA0 BUT AX  MAXTIN 01 OMEGA0 COS OMEGA0 TMAY HAVE NORM ARBITRARILY LARGE BY CHOOSING OMEGA0 TO BEARBITRARILY LARGE  THUS THE DIFFERENTIAL OPERATOR IS NOT BOUNDED ANDHENCE NOT CONTINUOUSENDEXAMPLESUBSECTIONTHE NEUMANN EXPANSIONLABELSECNEUMTHE NEUMANN EXPANSION PROVIDES